Data Center Interconnect

Data Center Interconnect refers to the technology used to link together two or more individual data centers to pool resources, balance workloads, replicate data, or implement disaster recovery plans – and providing data closer to the edge. Fiber optic DCI is deployed over short, medium, or long distances, typically ranging from 5km to 150km. A wide array of transport technologies is utilized to maximize bandwidth and minimize latency as cloud computing and the IoT continue to drive new levels of data center interconnect scale and complexity.

The unrelenting demand for storage and speed has led to a proliferation of new data centers around the world, including many in the hyperscale category with thousands of interconnects between them. With 256 QAM modulation, 400G can now be achieved on a single wavelength. This innovation provides a much-needed capacity boost, while fault detection, throughput, latency, and stress testing remain essential during data center interconnect installation. 

VIAVI supports the data center interconnect market from end-to-end by providing powerful, innovative solutions for the construction, commissioning, expansion, and monitoring of DCIs. This holistic approach encompasses robust performance verification over the DCI as well as the comprehensive testing of the essential equipment populating the hyperscale data center ecosystem.   

The term “hyperscale” has become synonymous with massive data centers housing countless servers within seemingly endless facilities. Cloud computing, internet content providers (ICP), and big data storage industries have indeed scaled their brick and mortar to unprecedented levels.

Hyperscale is increasingly used to define not just the scale and size of these new data centers, but also their architecture. Their size and scale are a useful place to start. Hyperscale data centers have a minimum of 5,000 servers and at least 10,000 square feet in size. In general, however, hyperscale data centers are much larger, frequently numbering tens of thousands of servers. Beyond the footprint and server figures, equally important is what is going on inside, where they’re architected for a homogeneous scale-out greenfield application portfolio using increasingly disaggregated, high-density and power-optimized infrastructures.

There are now more than 500 hyperscale data centers in the world today. By 2021, it is estimated they will number 628. By this time, they will represent 53% of all installed data center servers, 55% of all data center traffic, 69% of all data processing power and 65% of data stored in data centers.  They are monuments of advanced architecture, networking and automation.

A hyperscaler can and must provide nearly flawless service to millions of customers. Despite the advantages of customized computing, high speed networks, and advanced virtualization software occupying more real estate than many cities, no one data center can accommodate these demands independently. In the new hyperscale paradigm, data center connectivity solutions provide the binding element between all centers and enable a large conglomeration to behave as one entity.

Rapid deployment of DCI is an efficient method for maintaining network scalability and versatility without adding server capacity or square footage. Reliance on these vital data center interconnects for metro and long-haul linkage has increased vulnerability to external intrusions, accidental breaks, or disconnections that can lead to costly outages. For this reason, fiber monitoring solutions are essential tools for protecting DCI in a proactive fashion and minimizing mean time to repair (MTTR).

As the size and complexity have continued to develop, data center architecture has also evolved. Traditional three-tiered architectural models consisting of core routers, aggregate routers, and switches have been replaced by leaf/spine architectures to meet the demands of hyperscale data center solutions. Top of Rack (TOR) architecture reduces cabling by co-locating connections between switches and servers within the rack. End of Row/Middle of Row (EOR/MOR) architecture consolidates switch ports in designated locations and in some cases employ a patch panel as the central location for connectivity. 

With throughput requirements between data centers in the Terabit per second (Tbps) range, architectural solutions must also consider the data center interconnect configuration. Fiber infrastructure is selected based on the relative location, east-west traffic flow volume, and hardware configuration of each center. Extreme-density, high-fiber count cabling, 256 QAM and dense wave division multiplexing (DWDM) are essential options in this architectural mix. Long-haul DCI transport of 8o km or more also increases latency and may require EDFA or Raman amplification, or other active amplification elements that are not necessary for metro or campus deployments.

Although DCI is an integral component of data center architecture, it is also an independent entity with its own unique test and reliability considerations. The reach of DCI inside the data center is a brief yet eventful one. 

Advanced, rack-mounted data center interface solutions in the entrance room provide connection and access points for data center interconnect terminations. Interface cards within these devices introduce essential transponding, switching, and multiplexing functions that complete the DCI to data center handoff.

The compressed, densely populated hardware footprints found within most data centers are in stark contrast with delicate data center interconnect solutions that can extend for hundreds of miles. This combination of physical attributes and constraints creates unique data center test and certification challenges that can only be mitigated by a comprehensive portfolio of DCI test solutions.

Advanced fiber network certification solutions such as the VIAVI SmartClass Fiber OLTS-85  enable complete Tier 1 fiber network certification with a single compact device and provide test visibility of both the local and remote ends. The MPOLx is a full-function optical loss test set designed specifically for loss, polarity, end face inspection, and data center certification over native MPO connection interfaces.

Ever expanding customer demand has led to exponential growth in new hyperscale data center and DCI installations, carrying more complex workloads. This construction bonanza has mirrored a shift from the traditional stand-alone data center to the “mesh” network structure defined by increased leveraging of DCI. Testing strategies have likewise evolved to place more emphasis on automation, data center interconnect dependencies, fiber network security, and the aggregation of data center connections. 

To achieve 24/7/365 uptime, performance metrics like throughput, latency, and bit error rate (BER) must now be diligently tested and monitored both inside and outside the data center. The miles of connecting fiber between data centers also introduce countless new opportunities for breakdowns or disruption that can potentially impact millions of users. 

Problems can be hard to identify in the complex web of data center interconnect networks, so monitoring strategies and solutions must emphasize visibility and scalability. The rapid deployment of proprietary hyperscale data centers and DCIs has led to more interoperability protocols and infrastructure based on exotic or proprietary specifications that further complicate test solution development. 

A high volume of connections and pathways coupled with a high demand for uptime reliability makes testing of both the network and services essential in the data center interconnect environment. Hyperscale DCIs will often be expected to run continuously at full capacity. Monitoring solutions with little or no human intervention required are a logical fit for these environments, especially those that can be launched and controlled from virtual access points. The MAP-2100 rack-mounted test unit can execute  high-accuracy bandwidth throughput testing (RFC 2544, Y.1564) of data center interconnects to and from unmanned locations.

A successful DCI testing strategy will begin with an end-to-end (e2e) visibility strategy in the planning and deployment stages that propagates through the ongoing management and maintenance phases. Interoperability is a key attribute for test solutions in the DCI environment because test tools that support open APIs provide more flexibility and enable hyperscale organizations to establish their own automation code. 
As the hyperscale and data center interconnect environments advance their symbiotic transformation, VIAVI Solutions continues to work closely with standards bodies on open source initiatives and best practices that preserve independence and pathways to innovation. 

The challenges associated with DCI installation, activation, and monitoring require the best available fiber test solutions working collectively to ensure ongoing service quality. The rapid evolution of hyperscale technology combined with the high utilization factor of DCI pathways leaves little margin for error.

Fiber Inspection

Fiber is the lifeblood of data center connectivity solutions. Versatile fiber inspection solutions are an important element of data center interconnect verification and can prevent defects or contamination introduced during installation from degrading performance. VIAVI inspection solutions with automated routines, compact form factors, and available multi-fiber connectivity establish a firm foundation for DCI fiber connection assurance.   

High-Speed Transport of 400G and 800G

The bandwidth demands placed on data center networks ripple down to interconnects as they carry voluminous data over vast distances. Terabit Ethernet in the 400G and 800G categories is now among the transport technologies employed for data center interconnect that demand a more sophisticated level of BER testing and PAM-4 modulation support. Test solutions incorporating these cutting-edge technologies are invaluable during the development, deployment, and maintenance phases of the network lifecycle.

Metro 100G and 200G

DCI in the metro category typically falls between 5km and 80km and links data centers in nearby cities or mid-sized geographical areas. Metro interconnects often utilize 100G or 200G networking topology requiring fiber link qualification, characterization, and performance testing to ensure optimal performance. All-in-one network and fiber testing solutions utilizing automated test scripts add the requisite efficiency to turn-up, throughput, and high-accuracy latency test requirements.

Fiber Monitoring and Remote Fiber Test On Demand

Data center interconnects require consistent protection from intrusions, accidental outages, cuts, and breaks to ensure maximum uptime and minimal latency. Even a minor degradation in fiber quality can significantly reduce throughput and increase bit error rates.

With DCI capacity stretched to the limit, there is no allowance for compromised performance. The OTU-5000 or OTU-8000 rack-mounted OTDR test unit works in conjunction with the ONMSi remote fiber test system to form a proactive, automated solution for continuous data center interconnect monitoring. Fiber defects are located with pinpoint accuracy, triggering instant assessments, and alerts. Using a wavelength ranging between 1625-1650nm, the data center can be tested on demand while in service after a repair or routinely monitored with a very low scan time per fiber.  Remote, automatic fiber monitoring discovers and located fiber faults, allowing them to be tracked over time for the degradation that can cause latency and bit error rates. 

Preventative maintenance can prevent 20% of data center outages caused by the fiber plant. In the case of leased fiber, both parties have access to objective data that allows both to demarcate the fiber fault, repair it with the right team and to manage MTTR (Mean Time to Restoration/Resolution) key performance indicators.  This technique is critical to managing SLA contract terms for uptime as one-third of data center outages are caused by fiber plant problems on the DCI.

Typical data center interconnects will have two or three redundant, geographically unique fiber paths to provide resiliency and high availability. Both SmartOTU and ONMSi products can ensure visibility into the health of the data center interconnect, providing rapid notification when there is a threat from a break, bend, bad connector, and a dislodged/disconnected patch cord disrupting transmission.

Finally, using the high powered, modular OTU-8000, in high-sensitivity mode, one can automatically identify security threats from optical hackers who seek to take a copy of data center traffic. This is done by adding an optical coupler to the line and bending the fiber to capture a small amount of light. It is a low-cost way to access millions of dollars of data center traffic without detection. A fiber monitoring solution designed just for this use case with an ultra-high performance OTDR, and patented calculations is required to protect against this intrusion. Most data center interconnects carry regulated sensitive personal information data that must be protected against data theft and automated monitoring is the first line of defense.  Even if the data in encrypted, codes can be broken or stolen over time, so eliminating data theft is critical.

Virtual Test and Activation

Metro and core network mesh testing is greatly simplified through virtualized test agents. The software-based Fusion platform can be used to facilitate virtual service activation and performance monitoring via a combination of virtual test agents and hardware embedded in a DCI network. With the proliferation of data center interconnect across the data center landscape, these tools minimize technician dispatches and further enable rapid deployment, verification, and support SLA-protection.

Fiber Test and Certification

As bandwidth continues to expand, meeting customer requirements means multiplying fiber capacity. This translates to a higher volume and diversity of services, more end points/users and countless monetization opportunities. Fiber networks are an obvious solution to addressing bandwidth constraints. 

OTDR/Fiber Characterization and OSA test support the commissioning, construction, and in-service test of today’s high-capacity fiber optic interconnects. VIAVI offers the most comprehensive range of fiber testers for essential fiber optic test for this phase of install, maintenance, and troubleshooting with built-in automation for the full range of basic to detailed testing needs.

MPO

The compact footprint and fiber density inherent to Multi-fiber Push On (MPO) connectors are another option for exceedingly crowded hyperscale data centers and interconnects. Inspection and test tools providing a convergence of native MPO connectivity, automated workflows, and fast and reliable reporting allow the scalability and efficiency of multi-fiber connectors to be fully leveraged.