Peripheral Component Interconnect Express (PCIe) is the industry-standard high-speed computer bus architecture used to connect processors to peripherals, memory and other components. PCIe, now approaching its fifth generation with the anticipated release of PCIe 5.0, provides separate links to each connected device using a point-to-point topology.
With each successive iteration of PCIe, the speed and bandwidth have doubled, making the PCIe testing process more challenging as data capture, storage and visualization processes grow in complexity. Since the release of PCIe 1.0 in 2003, PCIe testing tools have continually adapted and improved to meet these challenges. Today, a full suite of PCIe test equipment is available to enable accurate and efficient PCI Express troubleshooting and testing.
- PCIe Test Challenges
While the original PCIe 1.0 had an available bandwidth of just 8GB/sec, the soon to be released PCI Express 5.0 will raise the bar up to 128GB/sec over 16 lanes of traffic. As the bandwidth increases, so does the potential for crosstalk and discontinuity, making recent innovations in PCB trace materials and lane margining a prerequisite for this rapid progression. The versatility of the PCIe format is another important factor, and the focus on interchangeability has influenced the designers of PCIe test equipment and hardware in equal measure. Backwards compatibility between PCIe revisions has remained a hallmark, and slot sizes of 1x through 16x are congruous with any size PCIe card, with the smaller of the two items dictating the bandwidth availability.
PCIe and NVMe
Non-Volatile Memory Express (NVMe) is the solid-state memory protocol developed specifically for use with PCI Express. “Non-volatile” in this context refers to memory that does not require power to retain data. NVMe has leveraged the PCIe architectural advantages to obtain read and write speeds more than twice as fast as a 12Gbps SAS interface. Instead of utilizing a traditional I/O controller, NVMe leverages a direct connection to the CPU via PCIe. In order to test the PCIe/NVMe combination effectively, PCIe test equipment with high-end memory capacity and segmentation features is integral to accurate NVMe traffic decoding.
The use of NVMe over Fabrics (NVMe-oF) has introduced additional PCIe test challenges by increasing the range between the NVMe storage device and host, while maintaining a goal of 10µs or less of additional latency introduction. Although the NVMe-oF specification is inherently similar to the NVMe base specification, characterizing the transport mechanisms adds additional complexity. Utilizing the appropriate PCIe/Ethernet/Fibre Channel Analyzer/Jammer test equipment at both the NVMe Enabled Host and NVMe Device ends of the circuit is a recommended best practice.
PCI Express 4.0
Released in October of 2017, PCI Express 4.0 is the current PCIe revision which has brought improvements such as lane margining and latency reduction to go along with the typical speed and bandwidth improvements inherent to each new PCIe generation. Extended tags and credits and enhanced I/O virtualization are additional improvements that will continue to be optimized with PCIe 5.0. The adoption of PCIe 4.0 has given added visibility to many of the PCI Express test challenges brought about by increased speed and resultant insertion loss and margining requirements. Traffic flow monitoring, data storage and error detection functions now require a higher performance standard from PCIe test tools, and this trend will continue unabated with PCIe 5.0 and beyond.
The adoption of PCIe 4.0 has given added visibility to many of the PCI Express test challenges brought about by increased speed and resultant insertion loss and margining requirements. Traffic flow monitoring, data storage and error detection functions now require a higher performance standard from PCIe test tools, and this trend will continue unabated with PCIe 5.0 and beyond.
The PCI Special Interest Group (PCI-SIG) defines the specifications and compliance testing required to guarantee interoperability of PCI Express systems. The group was formed in 1992 and now has over 800 member companies developing products based on PCI-SIG released specifications. These specifications are free to their member organizations or individuals. PCI-SIG also offers PCIe testing and compliance workshops. The PCI-SIG compliance library includes a comprehensive list of recommended equipment.
The PCI Express test categories included in the PCI-SIG specifications include electrical testing, configuration testing, link protocol testing, transaction protocol testing and platform BIOS testing. Channel topologies have become much more complex as the data rates have increased. Using simulation to optimize power and signal integrity is a recommended practice for PCIe link evaluation. Determining whether data packets are reliably transferred can be performed by utilizing protocol validation at the physical layer. Transaction protocol testing can debug unwanted errors and ensure PCI-SIG compliance.
- PCIe Test Tools
A wide array of PCIe test tools are now available. With each successive generation, PCI Express test equipment has evolved to meet the increasingly stringent demands. Feature-rich tools from industry-leading PCIe test equipment suppliers are ideally portable and rugged, with readily-available training and certification. Given the backwards-compatible nature of the PCIe interface, interoperability for multiple sizes and versions and rapid upgradability are other overall characteristics of outstanding PCIe test equipment.
A jammer can manipulate live traffic to simulate errors in real time. The term “jamming” originated in the early 20th century to describe the deliberate interference introduced to disrupt enemy communication lines. For PCIe testing, a jammer is an inline error injection tool that can simulate real-world conditions and shorten test cycles. Often using pre-defined automated test scripts, a jammer can recreate a wide variety of error-testing scenarios.
Jammers such as the Xgig are highly intelligent and protocol aware, and can utilize conditional jamming to maintain control over the test process and ensure comprehensive test coverage. Working in conjunction with a protocol analyzer or other PCIe tester, the jammer produces discernable triggers at the error injection points. By introducing errors into real-world environments, the responsiveness and efficacy of the error recovery process can be accurately discerned.
The protocol analyzer is a versatile PCI Express tool for bus throughput and link performance measurement as well as packet monitoring and recording. Additional triggering, error reporting and filtering features can enable rapid error identification. Jamming capabilities can artificially create latencies and retransmissions to exercise error-detection capabilities.
Powerful analyzers with advanced trace analysis, traffic flow visibility and memory segmentation features are invaluable. The Xgig 4K16 Protocol Analyzer/Jammer allows simultaneous error injection and protocol analysis. Interoperability features include lane width support for x1, x2, x4, x8 and x16. Users are alerted to errors at every layer of the stack and advanced memory utilization empowers the simultaneous capture of multiple traces.
PCIe test software is the backbone upon which protocol analyzer and jammer technology has continued to meet the increasing speed, functionality and versatility requirements. Well-designed software can automate repetitive functions, create customized routines based on released specifications and integrate multiple tools for seamless functionality. Advanced reporting software can simplify complex data analysis and facilitate optimized interpretation.
The Xgig Expert software provided with all Xgig analyzers creates a user-friendly interface that aids in rapid data interpretation, debugging and troubleshooting. The software efficiently sorts through captured events to characterize performance and interoperability issues, identify physical layer issues including protocol violations and organize results into comprehensive reports on demand. Additional reporting software can enable efficient data filtering, analysis and presentation capabilities.
- PCI Express Troubleshooting
Troubleshooting PCIe failures can often become a challenging task. Fortunately, many readily-available PCIe tools also provide exceptional debugging and troubleshooting capabilities. Keys to effective PCIe testing and troubleshooting include increased visibility of traffic flow and insight into the most commonly observed hardware issues. These issues include link speed problems such as equalization failures, traffic issues and quality issues observed after recovery. Error-reporting software can detect and log application-specific errors and characterize error types, such as time-outs and data transfer stops.
A protocol analyzer equipped with analytical software can become an important and effective PCIe troubleshooting tool. The analyzer will enable the easy view of traffic flow and historical data necessary to pinpoint issues and chronology. The analyzer also has powerful trigger conditions with the ability to capture extremely long sequences and filtering capabilities for specific packets that further enhance investigative prowess. The logging of time-ordered sequences facilitates troubleshooting by detailing events directly before and after reported errors.
- Valuable Possibilities with PCIe Test Equipment
Since the release of PCIe 1.0, the challenge of effective PCIe testing has been addressed through the ongoing development of a versatile assortment of PCIe test equipment and software. These tools have been designed to accurately measure performance, rapidly detect issues and simulate use conditions for robust and comprehensive analysis. New and innovative protocol/jammer combinations have proven to be the ideal complement for complex use conditions such as NVMe-oF.
Analyzers with a high level of visibility into traffic flow and advanced trace analysis will continue to be invaluable test tools with the upcoming release of PCIe 5.0. Jamming tools with the capability to manipulate live traffic create a robust level of simulation that otherwise would not be possible. The software behind these cutting-edge PCIe tools continues to tie it all together, with ever-improving interface and reporting enhancements that seamlessly unite the operator with the PCIe test case.
PCIe Architecture & Format
PCI Express vs NVM Express
Explore PCI Express 4.0 Speeds & More