Tag Archives: QSFP

Thermal Solutions Designed for 400G Ethernet

As noted in a recent article from Electronic Design, the IEEE 802.3bs Ethernet Working Group published its standards for 400 Gigabit Ethernet in December and at the DesignCon conference in Santa Clara, Calif. at the end of January many of the connector companies, including industry leaders such as Molex and TE Connectivity, were already rushing to create hardware to meet the new standards.

400G Ethernet

“The 8-lane, Quad Small Form-factor Pluggable Double Density (QSFP-DD) form factor is one of four connectors that will target 400G Ethernet connections,” Technology Editor William Wong explained. “The others include OSFP, CFP8, and COBO. These new form factors are needed to handle the 25-Gb/s NRZ, 56-Gb/s PAM-4, or potentially 112-Gb/s PAM-4 signaling needed for 400G and beyond.”

Hardware specifications were also defined by the Quad Small Form-Factor Pluggable Double Density (QSFP-DD) Multi-Source Agreement (MSA) group in Sept. 2017. In addition to Molex and TE Connectivity, the 62 companies that were part of the QSFP-DD MSA included Broadcom, Cisco, Finisar, Juniper Networks, and more leaders in the communications and broadcast industries.

The report said, “QSFP-DD supports up to 400 Gb/s in aggregate over an 8 x 50 Gb/s electrical interface. The cage and connector designs provide backwards compatibility to QSFP28 modules which can be inserted into a QSFP-DD port and connected to 4 of the 8 electrical channels.” From a thermal standpoint, the group defined the operating case temperatures as 0-70°C for standard class, -5°C to 85°C for extended class and -40°C to 85°C for industrial class.

“For all power classes,” the report continued, “all module case and handle surfaces outside of the cage must comply with applicable Touch Temperature requirements.”

At DesignCon, Wong wrote, one of the biggest challenges that companies faced was how to optimize thermal management for these optical systems. This is nothing new for designers of optical transceivers, as increased density means increased heat, which can lead to performance and reliability issues.

As a recent article from Advanced Thermal Solutions, Inc. (ATS) exploring industry developments in cooling QSFP transceivers explained, “The performance and longevity of the transceiver lasers depend on the ambient temperature they operate in and the thermal characteristics of the packaging of these devices. The typical thermal management approach combines heat dissipating fins, e.g. heat sinks, and directed airflow.”

It added, “Designers can reduce thermal spreading losses by keeping the heat sources close to the thermal interface area and by increasing the thermal conductivity of the case materials.”

ATS engineers have been working with QSFP manufacturers to meet thermal needs. In a recent project, ATS engineers designed a new solution for cooling optical transceivers using a series of heat sinks in an optimized layout that took into consideration individual QSFP junction temperatures as well as the effect of airflow pushing heat onto downstream transceivers.

400G Ethernet

Test set-up of different heat sink designs on QSFP28 connector cages. Through testing, analytical modeling and computer simulations, ATS engineers optimized a design for cooling a series of QSFP. (Advanced Thermal Solutions, Inc.)

Analytical modeling and computer simulations “showed that having heat sinks with fewer fins upstream and heat sinks with more fins downstream provided a near isothermal relationship between the first and last QSFP, an important consideration for QSFP arrangements.” Overall, the innovative design showed a 30 percent improvement over QSFP heat sinks currently on the market. Using this design, there was less than a degree difference between the first QSFP in the series and the last.

“Minimizing the upstream QSFP heat sinks, which in turn minimizes the amount of preheat to the downstream QSFP and allows as much airflow to enter downstream QSFP,” the engineers remarked. “At the same time ensuring the upstream QSFP temperatures are equal to or just lower than the downstream QSFP. This keeps the downstream QSFP temperatures at a minimum, while also keeping the transceiver stack close to isothermal.”

In addition to junction temperature, ATS engineers factored in contact resistance, since QSFP transceivers require that no thermal interface material (TIM) is used. As the engineers explained, “The QSFP isn’t fixed in the cage; it can be hot swapped. After a few insertions and removals, it will gunk up the TIM.” Also, to increase the surface area, the heat sinks were extended beyond the edge of the cage and the base had to be thicker than originally planned to account for spreading resistance.

As 400G Ethernet technology continues to expand, more power is required, and transceiver-density is increasing, ATS is ready with thermal solutions to meet next-generation challenges for the QSFP market.

For more information about Advanced Thermal Solutions, Inc. (ATS) thermal management consulting and design services, visit https://www.qats.com/consulting or contact ATS at 781.769.2800 or ats-hq@qats.com.

Industry Developments: Cooling QSFP Optical Transceivers

By Norman Quesnel
Senior Member of Marketing Staff
Advanced Thermal Solutions, Inc.

Rapid advancements in fiber optic technology have increased transfer rates from 10GbE to 40/100GbE within data centers. With the emergence of 100GbE technologies, the creation of data center network architectures free from bandwidth constraints has been made possible. The major enabler of this performance increase is the QSFP optical transceiver.

QSFP is the Quad (4-channel) Small Form-Factor Pluggable optical transceiver standard. A QSFP transceiver interfaces a network device, e.g. switch, router, media converter, to a fiber optic or copper cable connection as part of a Fast Ethernet LAN.

The QSFP design became an industry standard via the Small Form Factor Committee in 2009. Since then, the format has steadily evolved to enable higher data rates. Today, the QSFP MSA (multi-source agreement) specification supports Ethernet, Fibre Channel (FC), InfiniBand and SONET/SDH standards with different data rate options.

QSFP

Fig. 1. The Small QSFP Form Factor Allows More Connectors and Bandwidth than Other Fiber Optic Transceiver Formats. Note the Cooling Fins on Each Receiver Device. [1]

Thermal Issues

The small QSFP form factor has significantly increased the number of ports per package. The increased density of transceivers can lead to heat issues. The optical modules can get hot due to their use of lasers to transmit data. Even though the popular QSFP28 provides lower power dissipation than earlier transceivers – abut 3.5W, the QSFP28 factor has also allowed a significant increase in port density.

Newer microQSFPs can dissipate even more heat. microQSFP interconnects fit more ports (up to 72) on a standard line card, saving significant design space.

Fig 2. Air Gap Locations Shown in Thermal Specifications Feature on QSFP. Top: QSFP at the Inside Edge of a Cage, Bottom: QSFP Section Showing Typical Internal Layout. [2]

The performance and longevity of the transceiver lasers depend on the ambient temperature they operate in and the thermal characteristics of the packaging of these devices. The typical thermal management approach combines heat dissipating fins, e.g. heat sinks, and directed airflow.

Fig 3. Test set-up of different heat sink designs on QSFP28 connector cages. (Advanced Thermal Solutions, Inc.)

Recently, Advanced Thermal Solutions, Inc. (ATS) tested a variety of pin and fin-style heat sinks for their comparative cooling performance on a standard QSFP connector cage. For this setup, an even amount of heat was provided to each connector site via a heater block. Individual thermocouples measured the heat flux resulting with the different heat sink types.

A main goal of this test was how each of four heat sinks would perform while relying on airflow incoming from just one side. By the time it reached the fourth heat sink would the airflow provide enough conduction for adequate cooling? An image from this series of tests is below in Figure 4.

Fig. 4. Test Setup to measure cooling performance of individual heat sinks on a QSFP connector cage when airflow is from one side only. (Advanced Thermal Solutions, Inc.)

The tests results showed that the denser the heat sink pins or fins on the sink closest to the incoming air, the hotter the farthest away QSFP will be. Thus, the best solution used heat sinks whose pin/fin layouts were optimized to work in the actual airflow reaching them.

This meant more open layouts closer to the air source, allowing more air to reach denser pin/fin sinks farther from the air. The non-homogeneous heat sinks allowed for a low, uniform temperature across the QSFP for the most effective function of the QSFPs’ lasers.

microQSFPs

Cooling solutions are different between QSFP28 designs and microQSFP installations. QSFP28 transceiver cooling is typically provided at multiple connector sites. microQSFP modules, e.g. from TE Connectivity, have an integrated heat sink in the individual optical module. Used with connection cages that are optimized for airflow, their heat is controlled in high density applications.

Fig. 5. Integrated Module Thermal Solution (Fins) on microQSFPs Provides Better Thermal Performance and Uses Less Energy for Air Cooling. [3]

Fig. 6. A Video Demo from TE Connectivity Shows 72 Ports of microQSFP Transceivers Units Running at 5W Each and All Kept Under 55°C Temperature Using 82 CFM Airflow. [4]

Finally, another factor affecting cooling performance is surface finish and flatness. Designers can reduce thermal spreading losses by keeping the heat sources close to the thermal interface area and by increasing the thermal conductivity of the case materials.

For QSFP, the size of the cage hole for heat sink contact given in the multi-source agreement (MSA) can be increased giving a reduction in the thermal interface resistance and therefore module temperature.

References:
1. FMAD IO, http://fmad.io/images/blog/20160612-100g-connectors.png
2. https://arkansashq.wordpress.com/2016/10/11/pluggable-optics-modules-thermal-specifications-part-2/
3. microQSFP, http://www.microqsfp.com/
4. TE Connectivity, https://www.youtube.com/watch?v=k_qNj-yAKz4

For more information about Advanced Thermal Solutions, Inc. (ATS) thermal management consulting and design services, visit www.qats.com or contact ATS at 781.769.2800 or ats-hq@qats.com.