Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short]

🌈 Abstract

The article discusses an unexpected phenomenon where matrix multiplications on GPUs run faster when given "predictable" data, such as all-zero or all-one inputs, compared to more "unpredictable" data like random normal distributions. The author investigates this behavior and discovers that it is due to the impact of dynamic/switching power consumption in semiconductor chips, which can lead to power throttling and performance degradation when the input data causes excessive transistor switching.

🙋 Q&A

[01] Power Consumption and Performance

1. What are the two main mechanisms of power consumption in semiconductors? The two main mechanisms of power consumption in semiconductors are:

• Static/leakage power: The power lost by simply flowing power through the circuits, proportional to the amount of silicon that is powered.
• Dynamic/switching power: The power consumed when transistors rapidly switch states, which can significantly increase the overall power consumption.

2. How does the power consumption and performance of matrix multiplications depend on the input data?

• Inputs with more "predictable" data, such as all-zeros or all-ones, lead to lower dynamic/switching power consumption and higher performance, as there is less transistor flipping.
• Inputs with more "unpredictable" data, such as random normal distributions, lead to higher dynamic/switching power consumption and lower performance, as there is more transistor flipping.

3. How do power limits and clock speeds affect the performance difference between predictable and unpredictable inputs?

• As the power limit is decreased, the performance advantage of using predictable inputs increases, as the GPU becomes more power-constrained.
• As the clock speed is decreased, the performance advantage of using predictable inputs decreases, as both predictable and unpredictable inputs become limited by the clock speed rather than power.

[02] Marketing vs. "Real" Performance

1. How do Nvidia's marketed FLOPS numbers differ from the "real" performance that can be achieved?

• Nvidia's marketed FLOPS numbers are based on the theoretical peak performance, which assumes the GPU can sustain the maximum clock speed.
• However, in practice, the GPU may not be able to sustain the maximum clock speed due to power throttling, leading to lower "real" performance compared to the marketed numbers.

2. How does power consumption affect the difference between marketed and "real" performance?

• Power consumption is a crucial constraint, and as power becomes more limited, the "real" performance of the GPU can be significantly lower than the marketed FLOPS numbers.
• For example, the H100 GPU has a theoretical FLOPS advantage over the A100, but its "real" performance is often closer to 2x the A100 due to power throttling.

[03] Conclusion

1. What is the key takeaway from the article? The key takeaway is that the actual performance of matrix multiplications on GPUs can be significantly affected by the power consumption and power throttling behavior, which is influenced by the predictability of the input data. This highlights the importance of considering power constraints when evaluating and comparing the performance of different hardware.