Memory bandwidth
Also: Memory throughput
How fast a processor can move data between memory and compute, and in practice the number that decides inference and analytics performance more often than core count does.
Arithmetic is rarely the limit. Large-model inference and analytical queries both spend most of their time reading data, so a chip's useful speed is set by how fast it can be fed. This is why core counts mislead: a part with fewer cores and more bandwidth can finish first.
Measured rather than argued
DuckDB ran the full TPC-H workload at scale factor 1,000 against an Intel Xeon 6 with 96 cores and against NVIDIA's Vera, which has 88 Olympus cores and up to 1.2 terabytes per second of memory bandwidth. Vera's median composite score was 3,073,292.99 against 2,109,782.19, a 46 percent advantage with eight fewer cores. The reasons given were bandwidth for fast scans, low latency keeping cores occupied, and large caches holding the hash tables built during aggregations and joins.
Why it shapes hardware design
Accelerator designs are increasingly organised around feeding the compute rather than adding more of it, and the cost of that shows up as power. One wafer-scale part doubled its clock and reported roughly double the tokens per second per user, with a rack drawing 125 to 135 kilowatts against about 23 kilowatts for a single wafer of the previous generation.
Questions
- What does memory bandwidth mean?
- How fast data moves between memory and the processor, usually quoted in terabytes per second. For inference and analytics it is often the real limit, because those workloads spend most of their time reading data rather than computing.
- Does a higher core count mean faster?
- Not reliably. In DuckDB's TPC-H run at scale factor 1,000, a part with 88 cores beat one with 96 by 46 percent, and the reasons given were memory bandwidth, memory latency and cache size rather than the number of cores.