将5G实验基础设施集成至多站点NFV生态系统

摘要

本方案的目标是通过基于VPN的覆盖网络架构，支持将5G实验基础设施灵活地整合到多站点NFV生态系统中。此外，该方案还定义了如何验证集成的有效性，包括在具备NFV能力的小型空中飞行器上进行多站点垂直服务部署。

摘要

网络功能虚拟化（NFV）已被视为实现5^th 第五代移动通信网络（5G）的演进催生了一种新范式，该范式通过虚拟化技术将网络功能软件化，从而降低对专用硬件的依赖，简化网络功能的开发，并缩短部署周期与成本，进而支持电信服务及垂直行业应用的灵活部署。在此背景下，马德里卡洛斯三世大学（Universidad Carlos III de Madrid）、西班牙电信公司（Telefónica）与IMDEA网络研究所共同在5TONIC——一个专注于5G技术的开放式网络创新中心——构建了一个网络功能虚拟化（NFV）生态系统，能够在分布于不同地理位置、由多方提供的NFV基础设施之上，创建复杂且贴近真实环境的实验场景。本文介绍了在基于5TONIC的多站点NFV生态系统中接入新型远程NFV站点所定义的协议，详细说明了现有基础设施与新增基础设施的技术要求、通过叠加网络架构实现的互联互通方式，以及新站点接入所需执行的步骤。该协议通过将一个外部站点接入5TONIC NFV生态系统进行了实例验证。随后，协议进一步阐述了验证站点成功集成所必需的测试步骤，包括利用远程NFV基础设施部署一种结合小型无人驾驶航空器（SUAVs）的多站点垂直行业服务，以展示该协议在支持分布式实验场景方面的潜力。 关键词：第五代移动通信网络，5G，网络功能虚拟化，NFV，5TONIC，虚拟化，叠加网络，分布式实验，小型无人驾驶航空器，SUAVs

引言

自本 decade 初以来，第五代移动网络（5G）的引入意味着对电信行业进行彻底变革，要求电信运营商应对在 5G 框架下开发的新网络服务与应用所提出的更为严苛的技术规范^1,2。这些新规范包括但不限于数据速率的提升、无线传输延迟的优化以及运营成本的降低。在网络技术革新所依赖的各项基础技术中，网络功能虚拟化³（NFV）已成为推动这一代网络发展的关键技术之一。NFV 能够将传统上依赖专用硬件的网络功能通过通用物理设备（例如数据中心中的服务器计算机）实现软件化。借助这一新范式，电信运营商和垂直行业可将网络功能与服务部署为一组软件组件，从而在服务部署与维护方面节约成本，并显著提升网络基础设施的弹性。该方法减轻甚至消除了大多数网络功能及特定垂直行业功能对专用（通常更复杂且复用性较低）设备的需求，同时支持更高程度和更密集的操作自动化，进而降低部署与维护成本。

考虑到网络功能虚拟化（NFV）环境所能提供的各项优势，电信行业越来越多的相关利益方自然开始积极参与在NFV环境中测试新型服务构想。在此背景下，Telefónica与IMDEA网络研究所共同创建了5TONIC⁴，这是一个专注于5G技术的开放型研究与创新实验室。该实验室位于西班牙马德里，提供多种先进技术，供研究人员和合作伙伴推动5G服务的开发与验证。特别是，该实验室配备了一个实验性NFV平台，开发者可在符合ETSI标准的NFV生态系统中部署并测试其基于NFV的新应用与服务⁵。因此，设计选择与技术方案的实验性结论可在比生产网络更真实且灵活得多的环境中得出。该平台设计支持跨多个外部站点的实验活动，并可通过明确定义的协议灵活地与5TONIC互联。

5TONIC NFV 生态系统所采用的技术方案基于单一的 NFV 编排器，该编排器通过欧洲电信标准协会（ETSI）托管的开源 MANO（OSM）软件实现⁶。该组件负责管理与协调网络服务（NS）的生命周期。这些服务可由虚拟化网络/垂直功能（VNF）组合构建而成，并可部署于 NFV 平台所集成的任一站点。5TONIC NFV 生态系统的设计工作是在欧盟“地平线2020”计划 5GINFIRE 项目^7,8的框架下完成的，该平台曾支持在欧洲八个面向特定垂直行业的实验基础设施以及巴西一个通过跨洋链路连接的实验设施上开展的 25 项以上实验，这些实验通过竞争性公开征集方式遴选产生。此外，该平台还被用于在西班牙构建国家级的分布式 NFV 试验床，以支持西班牙 5GCity 项目^9,10内的实验活动。最近，平台进一步集成了另一个巴西站点，以支持巴西与欧洲之间科研创新合作框架下的联合示范活动（即 5GRANGE 项目^11,12）。最后但同样重要的是，该基础设施还被用于支持 5G-VINNI 项目范围内的第三方实验^13,14。NFV 平台的地理分布情况如图1所示。

有意组织若拥有自主的网络功能虚拟化（NFV）基础设施，可在获得5TONIC指导委员会批准后，灵活接入5TONIC NFV生态系统，成为分布式生态系统中的试验床提供方，并参与联合实验与示范活动。为此，这些组织必须具备符合OSM软件栈的虚拟基础设施管理器（VIM）。5TONIC NFV编排器能够与参与特定服务部署的各站点VIM进行交互，协调计算、存储和网络资源的分配与配置，以实现构成网络服务的虚拟网络功能（VNF）的实例化与互连，并控制其生命周期，从初始接入直至最终退役。

为了管理所有互连站点之间的控制和数据流量交换，5TONIC NFV 生态系统采用基于虚拟专用网络（VPN）的覆盖网络架构。该方法为集成到 5TONIC 生态系统中的外部站点提供基于公钥基础设施（PKI）的安全访问，支持 OSM 软件栈与分布在各个测试平台上的不同 VIM 之间交换 NFV 控制信息，同时也支持管理与配置所有 VNF 所需信息的交换。此外，该覆盖网络还支持在不同站点部署的 VNF 之间进行数据流量的传输。

在此背景下，本文详细阐述了一种将外部站点接入网络功能虚拟化（NFV）生态系统的协议。该协议假设整个生态系统由部署在中心站点的单一NFV编排器进行统一管理，且外部站点配备有符合该编排器软件栈要求的虚拟化基础设施管理（VIM）解决方案。所提出的协议能够灵活地引入新的NFV站点及面向特定垂直行业的基础设施，从而扩展实验生态系统的资源组合。这使得构建一个分布式的MANO平台成为可能，该平台可在单一NFV编排器的控制下，在多个站点上测试和验证新型网络服务及垂直行业应用。为说明该协议的内部运行机制，本文将以向现有的5TONIC NFV生态系统中添加一个外部NFV站点为例，具体描述外部站点和5TONIC所需部署的组件，以及集成过程中需执行的全部步骤。图2展示了此次集成的目标概览：基于NFV的新测试平台通过中心站点与其余外部基础设施之间的VPN连接，接入5TONIC平台，从而实现网络服务的部署。

此外，为了展示该协议的有效性，将基于5TONIC生态系统和一个配备支持网络功能虚拟化（NFV）的小型无人驾驶航空器（SUAV）的外部站点，演示一个简单的垂直服务部署。该垂直服务的设计灵感来源于Vidal等人⁹所提出的一项实验，但为便于本文说明已对其进行简化。图3概述了该服务，其目标是支持偏远地区的智慧农业活动。该服务设想由一个智慧农业服务提供商使用SUAV，收集并分发散布在农田中的气象传感器所产生的数据。为简化起见，本文所述实验仅考虑单个SUAV和一个传感器，该传感器可提供温度、湿度和气压测量值。在实验中，外部NFV站点托管一个Wi-Fi接入点，该接入点作为虚拟网络功能（VNF）部署在SUAV上。该VNF为传感器提供网络接入连接，并将感知数据转发至网关功能。后者作为VNF部署在地面设备（一台mini-ITX计算机）上。传感器至网关功能的数据分发采用基于消息队列遥测传输（MQTT）协议¹⁵的发布/订阅模式。网关功能对接收的数据进行处理后，将其转发至物联网（IoT）服务器。该服务器基于Mainflux¹⁶开源平台，作为VNF部署在NFV生态系统的中心站点。最后，该场景假设偏远地区通过蜂窝非3GPP接入网络提供互联网连接。因此，该服务包含两个额外的VNF：1）接入路由器VNF，实现连接至非3GPP接入网络的3GPP用户设备的用户面协议栈¹⁷；2）5G核心网的基础实现，支持在接入路由器与IoT服务器VNF之间转发信息。为此，5G核心网VNF提供了3GPP标准¹⁷所定义的非3GPP互连功能和用户面功能的用户面部分的简化实现。

最后，图4 展示了在方案开发过程中涉及的最重要流程，强调了这些流程之间的逻辑关联以及负责执行这些流程的实体。

方案

1. Provision of the central site of the NFV ecosystem (prior requisites of the experiment)

1. Allocate an IP address space to be used by the central site. For the purposes of this protocol, the private address space 10.4.0.0/16 will be used.
2. Install the Management and Orchestration (MANO) software stack in the central site. In particular, the experiment carried out throughout this protocol uses the Open Source MANO (OSM) Release SEVEN¹⁸, which requires the following resources: Ubuntu 18.04 as operating system, 2 Central Processing Units (CPUs), 8 GB of Random-Access Memory (RAM), 40 GB hard-drive-disk, and at least one network interface with Internet access. For the installation, follow the instructions available at the OSM Release SEVEN documentation¹⁸.
3. Set up a Virtual Infrastructure Manager (VIM) compatible with OSM in the central site. Specifically, the experiment uses OpenStack release Ocata²⁰, running on a Virtual Machine (VM) with Ubuntu 16.04 , 4 CPUs, 16 GB of RAM and 200 GB of hard drive. The NFV Infrastructure (NFVI) handled by this VIM comprises three server computers, each with Ubuntu 16.04, 8 CPUs, 32 GB of RAM and 2 TB of storage. For the installation, follow the Ocata release documentation²¹.
   1. Deploy a virtual network within the OpenStack cloud platform, using an IP address range from the address space allocated in step 1.1. This network, henceforth referred to as management network, will be used to support the exchange of NFV orchestration information between the OSM and the virtual network functions (VNFs) instantiated at the central site.
   2. Configure a virtual network (henceforth denominated as data network) to support inter-site data communications, between the VNFs of the central site and other VNFs executed at external sites. To this end, use an IP address range from the address space of step 1.1.
      NOTE: The implementation of the networks mentioned in steps 1.3.1 and 1.3.2 has been done using provider networks of OpenStack. Provider networks must be connected to the physical network infrastructure of the central site to guarantee an appropriate operation.
4. Connect both virtual private networks (i.e., the management and the data networks), as well as the VIM and the OSM machines, to an equipment providing edge routing functionalities. This router will serve as the entry point to the central site of the NFV ecosystem.
5. Make available a public experiment repository to provide all the content needed to carry out the experiment. In particular, this protocol uses the public repository at²².

2. Configuration of the virtual private network service

1. Allocate an IP address space to support the appropriate operation of the multi-site ecosystem, so that network communications can effectively be established among multiple sites.
   ​NOTE: Enabling effective network communications among multiple sites requires a careful design of the IP address space to be used by the NFV ecosystem, as well as by external sites that need to connect to it. In particular, the address space allocated for inter-site communications should not collide with the address space already in use at every other site for other purposes.
   1. Allocate an IP address space to be used by external sites. Addresses in this block will be assigned to NFV entities (e.g., VIMs) and VNFs of the external site. To exemplify this protocol, the private address space 10.154.0.0/16 will be used.
   2. Allocate an IP address space to the virtual links between the external sites and the NFV ecosystem. These virtual links will be supported by a VPN service. To exemplify this protocol, the address range 10.154.254.0/24 will be utilized for these virtual links.
2. Set up an equipment to provide the Virtual Private Network (VPN) service (i.e., a VPN server). In particular, the experiment uses a server computer with Ubuntu 16.04 (64-bit variant image), six independent CPUs, 16 GB RAM, 1 TB storage disk, and two network interfaces.
   1. Configure one of the network interfaces of the VPN server to allow the reception of connection requests from external sites through the Internet. To that end, it is necessary to use an interface of the server configured with a public IP address.
   2. Configure the link between the VPN server and the edge router of central site. In the experiment this link was allocated the address range 10.4.255.0/24. Configure appropriate network routes at the VPN server, so that the NFV ecosystem becomes accessible from external sites connected to the VPN service.
3. Install the VPN open-source software provided by the OpenVPN²³ project into the VPN server. Specifically, this experiment uses the OpenVPN version 2.3.10, and its deployment was done with the bash script "openvpn-install.sh", available at http://github.com/Nyr/openvpn-install (other installation options are described in the OpenVPN documentation²⁴). The bash script presents the alternative parameters that will result in the configuration of the VPN service.
   1. Select the IP address to listen to VPN connection requests (i.e., the public IP address).
   2. Decide which protocol (UDP or TCP) should be used to drive the communications over the VPN. In this case, the experiment leverages on UDP that is the recommended protocol.
   3. Specify the port that will comprise the duple (together with the public IP address) that will be used to receive the service connection requests. By default, the assigned value is 1194.
   4. Choose one of the DNS servers of the list prompted by the assistant that will handle the name resolution requests performed by the clients of the VPN service.
   5. Press any key to enable the automatic initiation of the VPN service installation process.
4. Edit the configuration file "server.conf" that is located under the "/etc/openvpn/server/" directory and include the "client-to-client" directive aiming to extend the basic setup provided by step 2.3. Thus, different clients connected to the VPN service will be able to reach each other.
5. Enable the individual client configuration within the VPN setup to be able to independently manage the routing assignments for each client.
   1. Add the "client-config-dir ccd" directive, editing the same configuration file as in step 2.4.
   2. Create the directory "ccd" using the command "mkdir /etc/openvpn/ccd/". This directory will serve during the next section of the protocol to place the files comprising the routing directives associated to the clients intended to be integrated within the platform.
6. Set up the firewall rules that are needed to allow the connections with the service, while protecting the VPN server against malicious attack. To that end, this experiment leverages on iptables²⁵, which is a command line utility developed to configure the Linux kernel firewall.
   1. First, block incoming traffic to the VPN server with the command "iptables -P INPUT DROP".
   2. Allow the reception of VPN connection requests with the commands "iptables -A INPUT -i <public-intf> -m state --state NEW -p udp --dport 1194 -j ACCEPT" (<public-intf> is the name of the VPN server interface with the public IP address) and "iptables -A INPUT -i tun+ -j ACCEPT".
   3. Allow traffic forwarding between the VPN server interfaces (i.e., the public interface and the virtual interface created by the VPN service called tun0), to enable the VPN server to process the service connection request. For this purpose, execute the command "iptables -A FORWARD -i tun+ -o <public-interface> -m state --state RELATED,ESTABLISHED -j ACCEPT && iptables -A FORWARD -i <public-interface> -o tun+ -m state --state RELATED,ESTABLISHED -j ACCEPT".
   4. Enable the VPN server to provide the network address translation (NAT) capability with the aim of supplying Internet access to the central site, executing: "iptables -t nat -A POSTROUTING -s 10.4.0.0/16 -o <public-interface> -j MASQUERADE && iptables -A OUTPUT -o tun+ -j ACCEPT".

3. Integration of an external NFV site

1. Obtain an appropriate IP address range to integrate the site into the NFV ecosystem. This address range will be provided by the network operations center of the NFV ecosystem. According to step 2.1.1 of this protocol, the experiment will use a range of IP addresses for the external site within 10.154.0.0/16.
2. Create and provide the security credentials to connect to the NFV ecosystem.
   1. Generate a VPN credential that will allow the new infrastructure to establish a secure connection with the VPN server. To this purpose, execute the command "bash openvpn-install.sh" in the VPN server, select the option "1) Add a new client" of the prompted list, and provide the name to be associated with that credential, e.g., uc3m_infrastructure. This step will generate a file with the VPN credentials (named "uc3m_infrastructure.ovpn" in the example).
   2. Create a text file in the "/etc/openvpn/ccd/" directory of the VPN server, including the routing directives (as specified in the OpenVPN documentation²⁴) that must be pushed by the VPN server each time a connection to the VPN service is established using the VPN credentials.
      ​NOTE: The name of the text file must match the name specified during the creation of the VPN credential (e.g., uc3m_infrastructure) to provide a customized configuration for every VPN client.
   3. Provide the VPN credential file to the technical staff of the external site. This must be done through a secure and reliable channel. In this experiment, a manual encryption process is used. To encrypt the VPN credential, execute the command "7za a -tzip '-p<password>' -mem=AES256 <encrypted-file> <credential-name>", setting <password> as the desired encryption key, <encrypted-file> as the chosen name for the encrypted file, and <credential-file> as the file name of the VPN credential file (e.g., uc3m_infrastructure.ovpn).
   4. Provide the encrypted credential to the technical staff of the new site, along with the key that allows the decryption procedures, through a secure communications channel.
      NOTE: In this experiment, the encrypted credentials were provided by electronic email, whereas the decryption key was sent through a separate channel, using the short message service (SMS), with an offline agreement of the telephone number.
3. Set up the environment at the new site, so as to establish the connection with the NFV ecosystem, and to allow the remote NFVI be attached to the OSM stack of the central site.
   1. Install the VPN software provided by OpenVPN²⁴ in a computer, to enable a virtual link between the external site and the central site of the NFV ecosystem. The computer with the OpenVPN software will serve as a VPN client or VPN endpoint at the external site. The virtual link will be realized by means of a protected VPN tunnel between the VPN endpoint and the VPN server. In the experiment, the VPN endpoint runs in a server computer with Ubuntu 18.04, 8 CPUs, 8 GB RAM, 128 GB storage disk, and 3 GbE interfaces (one for connecting with the VPN service over Internet).
   2. Activate IP forwarding in the VPN endpoint to support network routing capabilities. To that end, include the line "net.ipv4.ip_forward=1" in the system configuration file located in the "/etc/sysctl.conf" path, and load the updated configuration with the command "sudo sysctl -p".
   3. Decrypt the VPN credential file with the information received in step 3.2.4, using the command "7za e <encrypted-file>", where the <encrypted-file> is the file name of the encrypted VPN credential. Specify the decryption key when prompted by the command.
   4. Boot the OpenVPN software with the decrypted credential file using the command "sudo openvpn --config <credential-file>" (<credential-file> is the file name of the VPN credential). With this, the VPN endpoint will authenticate to the VPN server, and will automatically receive appropriate VPN configuration parameters and network routes. This way, the VPN endpoint will behave as an edge router with a virtual link to the central site of the NFV ecosystem.
   5. Verify the proper operation of the VPN endpoint, using the ping command to verify the availability of connectivity to one of the nodes of the central site (e.g., the OSM stack equipment).
   6. In the new site, select an OSM compliant VIM to allow operations with the MANO platform. For this experiment, OpenStack release Ocata is used.
      ​NOTE: OSM Release SEVEN supports the following virtual infrastructure managers: OpenStack, OpenVIM²⁶, VMware's vCloud Director²⁷, Amazon Web Service²⁸, Microsoft Azure²⁹, and Eclipse fog05³⁰ (see OSM documentation¹⁸ for specific configuration details).
   7. Install OpenStack release Ocata²⁰ (see the detailed procedures in the release documentation²¹).
   8. Deploy the NFV infrastructure in the external site and attach it to the VIM. In particular, this experiment uses an NFV infrastructure comprising three single board computers (SBCs), each with a compute capacity of 1 GB RAM, 4 CPUs, and 32 GB storage disk; and a single mini-ITX computer with 8 CPUs, 8 GB RAM and 128 GB for storage.
      ​NOTE: The external site exemplified in this protocol is based on an NFV infrastructure of NFV-capable small unmanned aerial vehicles (SUAVs) . The details followed to enable such infrastructure are provided in Nogales et al³¹. Steps 3.3.6 to 3.3.8 are optional, as an NFV infrastructure may already exist at the external site.
   9. Create an OpenStack project to specify the set of computational resources of the external site that will be integrated into the NFV ecosystem. To do so, access to the graphical user interface (GUI) provided by OpenStack, log in to the system with the administrator credentials, click on the + Create Project button of the Identity -> Projects tab, and create a project completing the displayed form with the requested information.
   10. Create a valid user that will manage the project created in the previous step. To this purpose, access the Identity -> Users tab with the same login as in the previous step, click on + Create User, and fill in the required fields of the displayed form (username and password), selecting the new created project as the primary project, and choosing the admin role.
   11. Modify the security rules to allow VNF communication permissions in the new site (in particular, enable SSH and ICMP traffic). To that end, access the OpenStack GUI with the credentials of the user created in the previous step, follow the sequence: Project -> Network -> Security Groups -> + Add Rule, and select the SSH option of the Rule drop-down. Repeat the process but selecting the All ICMP option included in the drop-down menu.
   12. Download the images of a trial service offered by the OSM community, the Ping Pong network service ("Fedora-x86_64-20-20131211.1-sda-ping" and "Fedora-x86_64-20-20131211.1-sda-pong") from the public experiment repository, and upload them to the VIM of the external site. To this purpose, follow the sequence Project -> Compute -> Images -> + Create Image, and create the images using the displayed form and selecting each of the image.
   13. Assign two IP address ranges within the address space of the external site (allocated in step 3.1). These ranges will be used to support the management of the VNFs of the external site and to enable inter-site data communication among VNFs, respectively.
   14. Create a provider network (control-provider) using the VIM. This network will support NFV communications between the OSM stack at the central site and the VNFs deployed at the new site for management purposes. This type of communications will also enable the OSM stack to configure VNFs after their deployment. To create a provider network in OpenStack, follow the sequence Admin -> System -> Networks -> + Create Network and fill in the details of the new network, using the selected IP address range in the previous step.
   15. Create a second provider network (data-provider) using the VIM. This network will support data communications among the VNFs of the site and other VNFs of the NFV ecosystem. To create this provider network in OpenStack, follow the sequence Admin -> System -> Networks -> + Create Network, and fill in the details of the new network using the assigned address range.
       ​NOTE: Instructions on how to create virtual networks will vary depending on the VIM software. Check their respective software documentation for details.
   16. Share the VIM-related information (in particular, the username/password, and the project created in the steps 3.3.9 and 3.3.10) with the technical staff of the central site, to enable the attachment of the VIM to the OSM software stack.
4. Attach the external NFV infrastructure to the OSM software stack of the central site, using the information obtained from the step 3.3.16.
   1. Verify the connectivity between the OSM stack of the central site and the VIM of the new site, using the ping tool.
   2. If the previous connectivity test is successful, attach the external VIM to the OSM stack of the central site. To do so, use the following command in the OSM machine: "osm vim-create --name <external-VIM-name> --user <VIM-username> --password <VIM-user-password> --auth_url <authentication URL> --tenant <project-name> --account_type <VIM-type>". In this command: <external-VIM-name> is the name selected to identify the VIM within the OSM stack, <VIM-username> is the name of the user authorized to handle the resources of the external site (see step 3.3.10), <VIM-user-password> is the password of the indicated user, <authentication URL> is the link to the API made available by the VIM to enable requests from the OSM stack , <project-name> is the project name defined in step 3.3.9, and <VIM-type> is the VIM software used (in this experiment, OpenStack).
5. Verify the appropriate attachment of the new VIM to the OSM stack of the NFV ecosystem.
   1. Execute the command "ro_id=$(docker ps | grep osm_ro | cut -d ' ' -f 1)” to identify the id of the container implementing the Resource Orchestrator (RO) module within the OSM system. This module is the responsible for interacting with the VIMs in order to coordinate and allocate the needed resources in the deployment of subsequent network services.
   2. Access the RO container using the command "docker exec -it $ro_id bash". This command utilizes the identifier obtained in the execution of the previous step.
   3. Check that the new VIM is included in the list of available datacenters, using the command "openmano datacenter-list". The new site should appear in the list with the same name as the previously introduced one in the step 3.4.2 with the <external-VIM-name> parameter.
   4. List the images that have been uploaded to the VIM of the external site, using the command "openmano vim-image-list --datacenter <external-VIM-name>". The <external-VIM-name> parameter indicates the name selected to identify the VIM within the OSM stack. If the execution of this command is successful, the connectivity with the external VIM has successfully been stablished. Check that the Ping Pong images are included in the list.
   5. List the networks available at the new site with the command "openmano vim-net-list --datacenter <external-VIM-name>". Check that control-provider and data-provider are present.
6. Perform a preliminary validation of the proper integration of the new site, using a trial service offered by the OSM community (all the content in this regard is included within the experiment repository). For this purpose, the commands included in the following steps will be executed in the equipment hosting the OSM stack.
   1. Onboard the VNF descriptors (VNFDs) to the OSM stack running the command "osm vnfd-create <vnfd-package>" for each of the VNFs composing the trial service (<vnfd-package> corresponds to the file name of the VNFD package).
   2. Onboard the NS descriptor (NSD) of the trial service with the command "osm nsd-create <nsd-package-descriptor>", where <nsd-package> indicates the file name of the NSD package (in this experiment, ping_pong_ns.tar.gz)."
   3. Start the instantiation of the Ping Pong Network Service (NS) on the external and the central sites, using the command "osm ns-create --ns_name <instantiation-name> --nsd_name ping_pong_ns --vim_account <external-VIM-name> --config '{vnf: [{member-vnf-index: '2', vim_account: <central-VIM-name>}]}'". The <external-VIM-site> parameter identifies the VIM of the external site within the OSM stack. The "--config" option indicates that all the VNFs composing the service must be deployed on the external site handled by that VIM, except the VNF identified by the index 2 in the NS, which will be deployed in the central site (the VIM of the central site is specified in the <central-VIM-name> parameter).
   4. Check that the NS has been deployed and its status using the command "osm ns-list". If the instantiation is successful, the status will change to "READY".
   5. Check the IP address of each of the two VNFs with "osm vnf-list" (necessary to log in to the machines afterwards).
   6. Connect to each VNF via SSH, using the command "ssh fedora@<VNF-IP>" (<VNF-IP> represents the IP address of the VNF to connect to, obtained in the previous step). Introduce the password "fedora" when prompted by SSH. Once logged into both machines, check their interfaces using the command "ip address show", and obtain the IP addresses on their interfaces attached to the data-provider network (interface eth1 in both VNFs). From one of the VNFs, perform a ping to the other VNF, using the remote IP address in the data-provider network. If there is connectivity, the preliminary validation test will be considered successful.

4. Validation of the NFV multi-site platform with a realistic vertical service

1. Download the VNF images from the public repository and upload them into the VIM of their corresponding site (see Figure 3), following the procedure detailed in the step 3.3.12. In particular, the external site will host the Access Point VNF, Router VNF, MQTT Gateway VNF, and Access Router VNF. The central site will host the 5G Core VNF and the IoT Server VNF.
2. Onboard the VNFDs and the NSD of the smart farming service to the OSM stack (all the descriptors can be downloaded from the experiment repository).
   1. Onboard the VNFDs to the OSM stack executing the command "osm vnfd-create <vnfd-package>", for each of the VNFs of the network service. In this case, the <vnfd-package> parameter corresponds to the file name of the VNFD package.
   2. Onboard the NSD to the OSM stack with the command "osm nsd-create <nsd-package>", where <nsd-package> indicates the file name of the NSD package (in this experiment, jove_uavs_scenario_nsd.tar.gz).
3. Deploy the smart farming network service. To this purpose, run the following command from the OSM command line interface: osm ns-create --ns_name <instantiation-name> --nsd_name jove_uavs_scenario_nsd --vim_account <external-VIM-name> --config '{vnf: [ {member-vnf-index: "5", vim_account: <central-VIM-name>}, {member-vnf-index: "6", vim_account: <central-VIM-name>} ], wim_account: False }'.
   NOTE: As indicated in the step 3.6.3., the <external-VIM-name> and <central-VIM-name> parameters indicate the sites where the VNFs are to be deployed. Particularly, all the VNFs composing the smart farming service will be placed into the new external site, except for those with index 5 and 6 (the 5G Core and the IoT server VNFs) that will be allocated to the central site.
4. Check that the NS has been deployed, following the same procedure as in step 3.6.4.
5. Access to the IoT server VNF with the command "ssh mosquittosubscriber@<VNF-IP>" and check its interface configured to communicate with MQTT Gateway VNF through the command "ip address show dev eth1". The IP address of the VNF (<VNF-IP>) can be obtained executing the "osm vnf-list" in the OSM command line.
6. Following an analogous procedure, access to the MQTT Gateway VNF, and run the command "sudo python3 publisher_MQTT_GW.py -ma <IoT-IP> -ba <MQTT-GW-IP>" where the <IoT-IP> is obtained in the previous step, and the <MQTT-GW-IP> executing the "ip address show dev eth1" command in the MQTT Gateway VNF. This step initializes the MQTT Gateway VNF, which will receive data generated by the sensor using the MQTT standard¹⁵, transmitting these data to the IoT server VNF using the same standard.
7. Prepare a Single Board Computer (SBC) attaching a meteorological sensor, and with transceiver capacity to transmit sensor readings towards the MQTT Gateway VNF.
   ​NOTE: To exemplify this protocol, an SBC model in particular has been used. Hence, the following steps may need to be adapted in case of utilizing a different SBC platform.
   1. Connect (e.g., using tin-soldered copper wires) the board pins of the sensor to the general-purpose input/output (GPIO) pins of the SBC, following the configuration scheme of Figure 5.
   2. Enable the I2C kernel module in the SBC to be able to verify if the sensor is detected. For this purpose, run the command "sudo raspi-config", follow the sequence Interfacing Options -> I2C -> Yes in the displayed menu, and reboot the SBC to make the changes effective.
   3. Verify that the sensor is detected Installing the software i2c-tools in the SBC, and executing the command "sudo i2cdetect -y 1". If so, a grid should appear indicating the position where the sensor is detected.
   4. Install the appropriate software libraries to allow the SBC reading and sending the data provided by the sensor. In particular, this experiment leverages on the RPi.bme280³² and paho-mqtt³³ Python libraries.
8. Using the mobile application of the SUAV, take off the aerial vehicle that hosts the Access Point VNF, and position it to provide wireless coverage to the SBC with the sensor.
   NOTE: The flight of the NFV-capable SUAVs are independent from the operational behaviour of the network service, which is able to operate whether the SUAVs are flying or in a state of repose to mitigate battery consumption. Thus, the step 4.8 is optional.
9. Attach the SBC in charge of reading the data collected by the sensor to the Wi-Fi wireless access point provided by the Access Point VNF). After a successful attachment, a wireless network path will be enabled from the sensor to the MQTT Gateway VNF.
10. Start the transmission of sensed data, running the command "python3 /home/ubuntu/sensorDataTransmission.py -a <MQTT-GW-IP>" in the SBC that incorporates the sensor (<MQTT-GW-IP> is the IP address obtained in the step 4.6.).
11. Access the web GUI provided by the IoT server VNF to check the correct real-time reception of the sensed data. To that end, check the IP address of the IoT Server VNF with the command "osm vnf-list", and type the following Uniform Resource Locator (URL) in a web-browser: http://<VNF-IP>:3001, where <VNF-IP> is the IP address of the IoT server VNF. Then, click on the Sensors Data Collection button of the Home tab, and verify the real-time update of the graphs included in the dashboard as data are received.
    NOTE: To be able to access to the URL mentioned in the step 4.12, the device with the web-browser attempting to reach that resource must be connected to the NFV ecosystem and have IP connectivity with the IoT Server VNF. The VPN service can be also used for this purpose.
12. Wait for an appropriate period of time to obtain representative results of the execution of the smart farming service. Then, collect the data stored in the IoT server VNF for further analysis. Considering that the sensor included in this experiment provides temperature, humidity and pressure readings every 5 seconds, the service in the experiment run for a period of 10 minutes, resulting in 180 samples of sensed data (60 for each meteorological value type).
13. Access the database of the IoT Server VNF to retrieve the sensed data for further analysis. To this purpose, execute the command "id_database=$(sudo docker ps | grep 'influxdb:' | cut -d ' ' -f 1)" on the IoT Server VNF, and then "sudo docker exec -it $id_database bash"
14. Export the data to a comma-separated value (CSV) file, running the command "influx -database 'mainflux' -execute "SELECT * FROM messages WHERE \"name\" = '<data>' " -format csv > /tmp/<filename>.csv". Modify the parameter <data> to select which type of sensed data is to be export with the "temperature", "humidity" or "pressure", and set the <filename> parameter to choose a name for the output file that will keep the results.
15. Save the data files generated in the previous step for later representation (see Representative Results section) and verification of proper operation of the smart farming service.

结果

在严格按照协议将新站点接入中央平台并运行一项网络服务以验证其正常功能后，图6展示了open-vpn-monitor工具的截图。可以看出，新站点正通过VPN进行其全部通信，表明其通信流量经由VPN传输，从而实现数据交换，并因此成功将新站点正确添加到VPN服务中。

如图3所示，该网络服务正将位于远程基础设施中的传感器信息传输至位于中心站点的服务器。此外，图7展示了通过OSM网页图形用户界面（GUI）成功部署网络服务的情况，表明该实验可通过位于中心站点的MANO架构，从中心站点在新的远程基础设施中正确实例化。此外，实验中完成服务部署所需时间约为八分钟。该时间值，加上将服务描述符导入编排平台所需的时间（约9秒，每个描述符约1.3秒，包括网络服务（NS）及各个虚拟网络功能（VNF）描述符），能够满足5G基础设施公私合作伙伴关系（5G Infrastructure Public Private Partnership）所规定的90分钟服务创建时间关键性能指标（KPI）³⁴。在此背景下，Vidal 等人⁹的研究包含了利用所提出协议在多站点环境下对服务创建时间的深入分析。

图8展示了从传感器收集的数据，包括湿度、温度和压力的数值。这些样本对应于传感器发送到位于5TONIC的远程服务器的所有数据，这些数值被存储在数据库中。所有这些数据表明，该平台能够在引入新基础设施后部署实际的网络服务，并能够正确实现站点之间的通信。

图1： VPN 服务站点分布。通过平台分布的 VPN 服务及其链路连接情况（所有连接均经过 5TONIC）。 请点击此处查看此图的放大版本。

图2．平台与VPN服务概览。 本图展示了平台的全部组成元素：中心站点及其NFV基础设施、VPN服务，以及新增并入系统的基础设施。同时展示了各元素之间的连接关系。请点击此处查看此图的放大版本。

图 3：网络服务概览。 该图展示了网络服务中涉及的各个组成部分，及其分布情况、逻辑连接和网络连通性。请点击此处查看此图的放大版本。

图4：协议工作流程。 每一列代表协议的一个部分，其中描述了所执行的每个操作、操作之间的逻辑关系以及负责执行该操作的组件。请点击此处查看此图的放大版本。

图 5：引脚配置示意图。该示意图展示了如何将传感器的板载引脚与集成该传感器的单板计算机（SBC）的 GPIO 引脚进行物理连接。请点击此处查看此图的放大版本。

图6： OpenVPN-monitor 快照。 该图显示聚合基础设施已连接至VPN服务，并展示了与其连接相关的一些详细信息。此外，该图还描绘了属于其他远程基础设施的额外连接。 请点击此处查看此图的放大版本。

图7：OSM NS部署状态。 OSM图形界面，显示测试网络服务在远程基础设施中的成功部署。请点击此处查看该图的放大版本。

图8： 传感器所采集数据的代表性分析。 （A）传感器每5秒周期性采集的温度数据示意图。（B）传感器每5秒采集的湿度数据的图形表示。（C）传感器每5秒采集的压力数据的可视化展示。请点击此处查看此图的放大版本。

讨论

此前所述协议最重要的方面之一是其出色的灵活性，能够将新的计算基础设施集成到网络功能虚拟化（NFV）生态系统中，而无论这些基础设施在地理上的分布情况如何（只要网络与远程站点之间的带宽和通信延迟能够满足要求）。这通过基于虚拟专用网络（VPN）的覆盖网络架构实现，该架构可建立虚拟链路，将远程站点连接至NFV生态系统的中心节点。该方法为支持NFV生态系统中各站点之间的NFV及数据通信提供了高效且安全的通道，降低了外部方访问和/或篡改与NFV编排流程及已部署服务数据相关的敏感信息的可能性。在此背景下，该协议还描述了一种特定方法，用于与将要集成新基础设施的外部站点安全共享VPN凭证。本协议以马德里卡洛斯三世大学、西班牙电信公司（Telefónica）和IMDEA网络研究所共同在5TONIC平台提供的NFV生态系统为例进行了说明，但该协议具有通用性，可应用于其他满足本协议第1步所述先决条件的NFV环境。

此外，值得强调的是，本方案的实施完全采用开源工具和软件。尽管某些专有解决方案可能提供潜在的有益功能（例如 Fortinet³⁵），但由于开源工具具备成本效益高、开源社区提供广泛软件支持以及高度可靠等固有优势，其使用显著促进了本方案所涵盖所有组件的集成。此外，开源技术的使用还能促进具有相似性质的组件之间的协同作用。例如，为了监控使用该平台的客户端的VPN连接状态，本方案中实现的VPN服务可依赖于open-vpn monitor工具³⁶（一种基于Python的监控工具，能够与OpenVPN服务器实现互操作）。

另一方面，该协议规范考虑了在不同站点上实例化网络服务以进行验证。在此方面，需要强调的是，在特定站点上部署服务取决于该站点的计算、存储和网络资源的可用性，以及部署过程中可能需要的专用设备（例如，支持网络功能虚拟化的SUAV）。这并非协议本身的限制，而是对有意复现本文所述实验的利益相关者应予以考虑的因素。

此外，需要注意的是，部署网络服务所需的时间在很大程度上取决于多个因素，例如编排器与各个VIM之间的网络路径、VIM与其所管理的计算节点之间的数据通信性能，以及这些计算节点的内在特性（不仅与其可用的计算资源有关，还与其用于实现网络功能虚拟化的技术相关）。

最后，鉴于该平台及其VPN服务在迄今为止所参与的欧洲项目和协作工作中表现出的卓越性能（例如本文引言中提到的5GINFIRE、5GRANGE和5GCity），该平台将被视为马德里卡洛斯三世大学、西班牙电信公司和IMDEA网络研究所参与的新兴欧洲项目（如“地平线2020”计划中的LABYRINTH项目）以及国家级项目（如TRUE-5G）中的重要组成部分。

材料

本文使用的材料清单

姓名	公司	目录编号	评论
Bebop 2	Parrot		实验中用于运输树莓派（RPis）的无人机（UAV），从而为外部站点的计算单元提供移动性。
BME280 传感器	Bosch		可提供环境温度、大气压力和湿度读数的传感器。
商用 Intel Core Mini-ITX 计算机	Logic Suppy		用于托管实验外部站点中 OpenStack 控制节点（以虚拟机形式运行）的计算机服务器。此外，该设备的另一台单元（与树莓派一起）构成了该站点内 NFV 基础设施的计算资源。
Iptables	Netfilter - 开源工具		（软件）一个开源的命令行工具，用于配置 Linux 内核防火墙规则集。源代码可在线获取：https://www.netfilter.org/projects/iptables/
锂电池扩展板，型号 KY68C-UK	Kuman		为构成外部站点 NFV 基础设施的无人机计算单元供电的 HAT（顶部附加硬件）电源模块。
MacBook Pro	Apple		实验中使用的通用笔记本电脑，用于按照文稿描述获取和收集实验结果。
Mainflux	Mainflux Labs - 开源平台		（软件）实验中使用的开源物联网（IoT）平台，用于实现名为 IoT Server VNF 的虚拟网络功能。此外，该平台包含基于 Grafana 的开源软件，可用于可视化和格式化度量数据。源代码可在线获取：https://www.mainflux.com/
开源 MANO（OSM）- 版本 FOUR	ETSI OSM - 开源社区		（软件）实验中配置的 NFV 系统的管理与编排（MANO）软件栈。源代码可在线获取：https://osm.etsi.org/docs/user-guide/
OpenStack - 版本 Ocata	OpenStack - 开源社区		（软件）用于在实验中搭建中心站点和外部站点 NFV 基础设施的开源软件。源代码可在线获取：https://docs.openstack.org/ocata/install-guide-ubuntu
OpenVPN - 版本 2.3.10	OpenVPN - 开源社区		实现实验中所述虚拟专用网络（VPN）服务的开源软件，用于创建支持 NFV 生态系统运行的覆盖网络（为生态系统中所有站点提供互联）。源代码可在线获取：https://openvpn.net/
Openvpn-monitor	Python - 开源软件		（软件）基于 Python 代码的开源程序，用于可视化 VPN 服务状态，并展示每一时刻连接的站点。该程序通过定期检查由 OpenVPN 实现的 VPN 服务器提供的信息来实现上述功能。源代码可在线获取：https://github.com/furlongm/openvpn-monitor
Paho-mqtt 1.5.0	Python - 开源库		（软件）使用 Python 开发的开源库，支持通过 MQTT 标准传输传感器读取的数据。  源代码可在线获取：https://pypi.org/project/paho-mqtt/
Ping	Debian - 开源工具		（软件）一种开源测试工具，用于验证通过通信网络连接的两个设备之间的连通性。此外，该工具还可评估网络性能，因其可计算往返时间（即发送和接收一个数据包所需的时间）。  源代码可在线获取：https://packages.debian.org/es/sid/iputils-ping
Power Edge R430	Dell		高性能计算机服务器，为实验中所述的中心站点提供计算能力。
Power Edge R430	Dell		负责托管虚拟专用网络（VPN）服务的高性能计算机服务器。请注意，由于该服务中加密操作的资源消耗较高，因此对计算资源的需求也较高。
Power Edge R630	Dell		用于托管执行 MANO 软件栈的虚拟机（VM）的设备。此外，中心站点的 OpenStack 控制节点也作为虚拟机在此设备上运行。请注意，该设备的使用并非绝对必要，因为上述虚拟机对资源要求不高，因此也可由性能较低的设备完成相应操作。
树莓派，型号 3b	Raspberry Pi Foundation		所选的单板计算机（SBC）型号，用于为实验的外部站点提供计算能力。此外，该 SBC 型号在部署所包含的真实服务时，用于解释和发送传感器采集的数据。
RPi.bme280 0.2.3	Python - 开源库		（软件）使用 Python 开发的开源库，用于与 Bosch BME280 传感器进行接口通信，并解析该传感器提供的读数。源代码可在线获取：https://pypi.org/project/RPi.bme280/