IP + Optical: The Mainstream Solution for the 400G Era

A guest post by Italo Busi, IETF Standards Expert, Huawei

1. Traffic growth drives IP and optical transport networks into the 400G+ era

With the development of 5G and fiber to the home (FTTH), network traffic is increasing rapidly. According to Omdia‘s predictions, the annual growth rate of network traffic will reach more than 30% after 2021. This requires the IP network and optical transport network to provide higher bandwidth and lower per-bit cost.

How can we provide higher network bandwidth? From the perspective of IP and optical industry development, there are two major trends.

1.1 IP device ports will continuously speed up from 100G to 400G or higher

As shown in the following figure, 400GE, 800GE, and 1.6TE actually share the four-level pulse amplitude modulation (PAM4) industry chain. In the short-haul transmission scenario of less than 10 km, both 400GE and 800GE use the single-channel signal transmission rate of 50 Gbit/s. The only difference is that 400GE integrates four optical signals and 800GE integrates eight optical signals.

According to LightCounting, 800GE will surpass 400GE in sales and become the mainstream port rate of routers and switches by 2024.

1.2 As IP networks change in architecture and work with optical transport networks, one-hop connections become the mainstream choice

The IP network architecture is changing from a ring topology with hop-by-hop forwarding to a tree topology with one-hop connection. The new network architecture introduces an optical transport network with mesh connections to effectively improve the bandwidth of the entire network even though the port rate of IP devices is not increased. This network architecture change is extending from the backbone network to the metro aggregation layer and even the access layer.

When constructing future-oriented target networks, global operators use both architecture solutions to effectively reduce the network-wide TCO. At the backbone network and metro core layer, router ports will generally speed up from 100GE to 400GE, and optical transport networks will speed up from single-wavelength 200G to single-wavelength 400G or even 800G. At the metro aggregation and access layers, router interfaces will speed up from 10GE to 100GE, and optical transport networks will speed up from single-wavelength 10G to single-wavelength 100G or even 200G.

2.  IPoWDM has failed in the 100G era and cannot be implemented in the 400G era

During the synchronous acceleration of both IP devices and optical transport devices, some device vendors proposed the IP over WDM (IPoWDM) solution. As shown in the following figure, this solution does not require gray optical ports on IP routers and WDM devices. Colored optical ports account for most of the optical port costs.

The IPoWDM solution saves a little in cost, but has obvious defects in IP board integration, network O&M, and networking architecture. When the rate of router interfaces increased from 10GE to 100GE, 100G IP colored optical ports appeared for the first time in the industry around 2013. After ten years, almost no operator in the industry has used this solution to build their networks.

Today’s router interfaces are now faster, accelerating from 100GE to 400GE. With the mature commercial use of 400G ZR+ optical modules, IP colored optical boards and gray optical boards have almost the same integration level. Therefore, some device vendors believe that the IPoWDM solution with 400G ZR+ will replace the IP+WDM solution and become the mainstream network construction solution for IP network speedup.

However, according to the development history of the IPoWDM solution in the 100GE era, the 400G ZR+ solution does not resolve the four inherent risks of IPoWDM and cannot become operators’ mainstream choice. The following figure shows the four inherent risks.

• Limited network evolution.
• Loss of network openness.
• Higher network TCO.
• More complex network O&M.

2.1 Limited IPoWDM network evolution

The IPoWDM solution is extremely limited in terms of single-wavelength acceleration and single-fiber capacity improvement.

2.1.1 IPoWDM is restricted in the 400G era and is difficult to evolve to 800G or 1.6T

In terms of single-wavelength acceleration, the IPoWDM solution will be restricted in the 400G era for the long term, while the IP+WDM solution is easy to evolve to 800G or even 1.6T.

As shown in the following figure, 400GE IP gray optical ports and IP colored optical ports (ZR+) have used QSFP-DD optical modules in the 400G era. ZR+ colored optical ports have poor transmission performance and can only be used in metro networks. QSFP-DD modules are restricted by power consumption standards and cannot be developed to 800G ZR+ in the future. QSFP-DD modules may have to be replaced with CFP2 modules whose integration is several times higher than that of QSFP-DD modules. Even if QSFP-DD modules are replaced with CFP2 modules, the 800G ZR+ performance is lower than the 400G ZR+ performance. The ZR+ IPoWDM ports with higher rates cannot be implemented.

Moreover, IP gray optical ports and WDM ports of optical transport devices are accelerating rapidly. IP gray optical ports will speed up to 800GE or even 1.6TE in the next 2 to 3 years. The co-packaged optics (CPO) mode can be used to integrate an optical module into a switching chip in the next 5 to 6 years, and the rate of a single port will quickly increase to 2 Tbit/s or higher. In this case, the integration of IPoWDM colored optical ports is lower than that of gray optical ports. The IP+WDM solution does not require the same integration of gray and colored optical ports. Common WDM ports can use CFP2 or MSA optical modules with higher power consumption. Currently, the single-wavelength capacity is 800 Gbit/s. The single-wavelength capacity will be 1.2 Tbit/s in the next year and will increase to 1.6 Tbit/s in the next 3 to 5 years.

2.1.2      The single-fiber capacity of IPoWDM is restricted to 25.6T

The single-fiber capacity improvement of the IPoWDM solution will be restricted to 25.6T (64 x 400G) for a long time. In contrast, the IP+WDM solution now supports 96T single-fiber capacity and can be easily evolved to 144T in the future.

As shown in the following figure, the IPoWDM solution supports only the mature C96 fiber spectrum \ and lacks the capability of E2E integration with an optical transport system. The tunable lasers, modulators, and receivers used by IPoWDM optical modules cannot drive the industry chain to expand the fiber spectrum. WDM optical transport device vendors, however, have the capability of E2E integrated R&D of WDM optical modules, ROADM modules, and optical amplifiers, and can expand the fiber spectrum through continuous investment. Currently, the WDM optical transport system supports C-band 120 wavelengths and will be expanded to L-band 120 wavelengths in the next year. The S-band 120-wavelength system has also been prototyped.

2.2  Loss of IPoWDM network openness

IPoWDM only integrates L1 to L3. In fact, an extra L0 active device needs to be configured to install components such as ROADM modules and optical amplifiers. In the IP+WDM solution, L2 and L3 functions are integrated on IP routers, and L0 and L1 functions are integrated on WDM optical transport devices. Compared with the IP+WDM solution, the IPoWDM solution will lose openness in three aspects.

2.2.1 IPoWDM cannot implement interconnection between IP colored optical ports or between L0 optical-layer devices

The IPoWDM solution cannot implement interoperability between IP colored optical ports or between L0 optical-layer devices from different vendors. As a result, the entire network market will be occupied by a single vendor. If multiple vendors are introduced, the interconnection cost will greatly increase.

• The first issue is interoperability between IP colored optical ports. 400G ZR optical modules with standard oDSP modulation and FEC algorithms can be used as 80 km long-haul gray optical ports for interconnection. However, compared with ZR optical modules, ZR+ optical modules introduce a large number of non-standard modulation and FEC algorithms to improve transmission performance. As a result, IP colored optical ports using 400G ZR+ optical modules cannot communicate with each other.
• The second issue is the interoperability between L0 optical-layer devices. Generally, L0 optical-layer devices use optical supervisory channels (OSCs) for communication. Currently, the OSCs of optical transport devices from different vendors cannot be interconnected. In addition, many operators use WSON protocols to implement protection against fiber cuts, and the WSON protocols of different vendors are not compatible.

In the common metro network shown in the following figure, the IPoWDM solution is used, and the IP market and WDM market are locked together; i.e., a maximum of two vendors can be selected for the three network layers. In addition, two sets of IP devices need to be deployed at the aggregation layer to implement interconnection between two vendors’ devices, resulting in great waste. Moreover, all services must be converted into IP packets before they are transmitted. As a result, native service interfaces such as storage FC and video SDI interfaces cannot implement lossless transmission.

In the IP+WDM solution, mature gray optical Ethernet/OTN interfaces are used on IP routers and WDM optical transport devices. Therefore, there is no interconnection issue.

In the metro network shown in the following figure, the IP+WDM solution is used, five vendors can be selected for the three network layers, and only one IP device needs to be deployed at the aggregation layer to implement interconnection. Independent WDM optical transport devices can implement lossless transmission of native services such as storage FC and video SDI services.

2.2.2  IPoWDM incurs huge multi-vendor integration workloads for operators

The IPoWDM solution forcibly isolates L0 and L1 in the WDM optical transport system, which brings demanding requirements for integrated software development to operators.

• First, operators need to develop multi-vendor integration management software to provide functions such as network monitoring and fault-location management, service provisioning and the commissioning controller, and network planning and design. Open SBIs, such as NETCONF+YANG, need to be used to interconnect with L0 optical-layer devices and L1 IP colored optical devices. This will increase development workload by millions of lines of code. The IPoWDM solution also requires operators to have powerful software and hardware development teams and be able to troubleshoot code faults for different vendors.
• Second, the WDM optical transport system is a simulation system. End-to-end performance optimization depends on accurate physical-layer modeling and commissioning algorithms, and requires in-depth understanding of optical-layer parameters and extensive actual deployment experience. These are capabilities that operators do not have. When an operator integrates the L0 ROADM devices and L1 Xponder devices of multiple vendors, the performance of the entire network is greatly degraded compared with the performance of a network with single-vendor integration. Generally, the performance degradation can reach more than 5 dB.

2.3 Higher TCO for IPoWDM networks

The IPoWDM solution reduces network TCO. However, the TCO of the IPoWDM solution is higher than that of the IP+WDM solution in most scenarios, as explained below.

2.3.1 IPoWDM only has 400G ZR+ ports, which cannot meet traffic and transmission distance requirements in different scenarios

Currently, the IPoWDM solution provides only 400G ports, whose TCO is lower than the total TCO of 400GE IP gray optical ports and 400G WDM ports.

However, in actual networks, IP devices need to reduce the per-bit cost based on traffic requirements, and the port rate needs to match the actual traffic. WDM devices need to reduce the per-bit cost based on the transmission distance requirements, and the port rate needs to match the transmission distance without regeneration.

For example, in the backbone network scenario, when the transmission distance exceeds 2,000 km or if fiber attenuation is large, the IP+WDM solution can use single-wavelength 200G WDM ports to meet transmission distance requirements without regeneration. OTN switching can be applied to inversely multiplex services of IP gray optical ports at any rate, such as 400GE/800GE/1.6TE, to N single-wavelength 200G signals for transmission. In this way, the TCO of the entire IP+WDM network will be lower.

In metro short-haul DCI scenarios within 80 km, the traffic of a single site generally exceeds 400 Gbit/s. The IP+WDM solution can use single-wavelength 800G or even 1.2T/1.6T WDM ports to meet the transmission distance requirements without regeneration, and can also apply OTN switching to multiplex N 400GE services of IP gray optical ports into single-wavelength 800G/1.2T/1.6T signals for transmission. As a result, the TCO of the entire IP+WDM network is still lower.

At the metro aggregation and access layers where the actual traffic is low, the traffic of a single site is generally 100 Gbit/s or even lower than 50 Gbit/s. The IP+WDM solution can use single-wavelength 25G/50G/100G WDM ports to meet different traffic requirements of sites. In this case, IP devices can be flexibly connected to WDM devices through 10GE, 50GE, or 100GE ports to ensure the optimal TCO of the entire network.

2.3.2 400G ZR+ transmission performance is poor & the TCO of IPoWDM is still high when single-site traffic is 400 Gbit/s

The 400G ZR+ ports provided by the IPoWDM solution have poor transmission performance and support only about 240 km transmission on a backbone network.

On a metro network, services can pass through a maximum of three ROADM sites. With the Huawei IP+WDM solution, however, services can pass through a maximum of 12 ROADM sites through 400G WDM ports. As a result, when the single-site traffic is about 400 Gbit/s, the TCO of the IP+WDM solution is still lower.

As described in the following table, in the most common backbone, metro core–aggregation, and aggregation–access networking scenarios of telecom operators, the IPoWDM solution with 400G ZR+ cannot provide a one-hop connection at the optical layer due to poor performance. As a result, TCO is higher than for the IP+WDM solution. The TCO of the IPoWDM solution can be lower only in scenarios where metro DCI is implemented and where the single-site traffic is about 400 Gbit/s. When the single-site traffic reaches 800 Gbit/s or higher, the IP+WDM solution can use single-wavelength 800G/1.2T/1.6T WDM ports to achieve lower TCO.

2.3.3 IPoWDM does not support optical-layer WSON protection and has higher TCO for network-wide protection

In the IPoWDM solution, only IP devices can be used to implement network-level protection. Therefore, sufficient electrical-layer resources (such as interface boards and switching capacity) of IP devices must be reserved to protect services against multiple fiber cuts.

In the IP+WDM solution, both IP and WDM devices can be used to implement network-level protection. The low-cost ASON/WSON solution can be used on WDM devices to protect services against multiple fiber cuts as long as some fiber spectrum resources are reserved. Currently, global mainstream operators choose the IP+WDM hybrid network-level protection solution. For example, the port dual-homing solution of IP devices is used to protect services against IP device or board failures, and the ASON/WSON solution of WDM devices is used to protect services against fiber cuts, reducing overall TCO.

2.4  More complex O&M on IPoWDM networks

In the 100GE era 10 years ago, the IPoWDM solution exposed obvious defects in network O&M.

1. Compared with the IP+WDM solution, the IPoWDM solution has unclear network layers, making fault demarcation and analysis difficult. Especially for irregular fault events, it is difficult to determine whether they are caused by the IP protocol or optical-layer OCh/OMS performance. Fault responsibilities are also not clear. Even if the colored optical router and L0 ROADM are provided by the same vendor, responsibilities cannot be clearly determined for the two types of devices if a fault occurs

2. As shown in the following skill map, L3 IP and L0+L1 WDM belong to different industry systems. They have their own complex technologies in terms of design principles, configuration principles, commissioning principles, and O&M functions. It is difficult for an O&M engineer to master the technologies of both systems. Even if some operators forcibly merge the planning, construction, and O&M of IP and WDM devices into one department, the two types of devices need to be maintained by different engineers. Therefore, the fault demarcation issue of the IPoWDM solution cannot be resolved. This is one of the main reasons why the IPoWDM solution failed to develop in the 100G era.

3. Summary

The IPoWDM solution cannot resolve the four major risks: difficult network evolution, loss of openness, higher TCO, and more complex O&M. In the 400G era, the IPoWDM solution with innovative ZR+ technology can be used only in some metro short-haul DCI scenarios. In most other scenarios, the IPoWDM solution has obvious disadvantages.

The IP+WDM solution will always be the mainstream choice for the long-term development of operator networks.

Disclaimer: Any views and/or opinions expressed in this post by individual authors or contributors are their personal views and/or opinions and do not necessarily reflect the views and/or opinions of Huawei Technologies.