NVIDIA GB300 NVL72: A leap in AI performance and efficiency



Key features and technological breakthroughs of the NVIDIA GB300 NVL72

Rack-Scale “AI Factory” Design: The GB300 NVL72 is a single 72-GPU, 36-CPU platform in a rack-scale form factor, meaning all components are integrated within one rack for maximum density. The entire system is 100% liquid-cooled, allowing it to safely dissipate the heat of a ~120–130 kW rack and enabling much higher power density than air-cooled setups. This liquid cooling is crucial for modern AI data centers – it supports more GPUs per rack, improves energy efficiency, and prevents thermal throttling, ultimately yielding greater performance in a smaller footprint. In practical terms, liquid cooling lets the GB300 NVL72’s high-wattage GPUs run at full tilt continuously, which is essential for AI factories and cloud deployments.
Blackwell Ultra GPUs with Enhanced AI Compute: At the heart of the platform are 72 NVIDIA Blackwell Ultra GPUs, representing NVIDIA’s latest GPU architecture (Blackwell). The “Ultra” variant is an especially powerful model: each Blackwell Ultra chip combines two reticle-sized GPU dies and is loaded with 288 GB of HBM3e high-bandwidth memory, delivering roughly 15 petaflops of FP4 AI performance per GPU. Compared to standard Blackwell GPUs, the Ultra GPUs feature 1.5× more AI compute FLOPS and specialized optimizations for inference. Notably, the Tensor Cores in Blackwell Ultra are “supercharged” with 2× faster Transformer attention processing and new low-precision modes like FP4, which significantly boost throughput for large language models and reasoning tasks. These GPUs enable the platform to achieve exaflop-class performance – over 1.1 exaFLOPS (1,100 petaflops) of tensor processing at FP4 precision (or up to 1.4 exaFLOPS with sparsity) across the whole rack. This represents a massive compute leap, empowering larger and more complex AI models than ever before.
Grace CPUs and Unified Memory: Complementing the GPUs are 36 NVIDIA Grace CPUs, based on Arm Neoverse V2 cores, which provide powerful CPU throughput and memory bandwidth to feed the GPUs. Each Grace CPU features 72 cores, and together the 36 CPUs contribute 2,592 cores in the system, all optimized for AI and data center workloads. The Grace CPUs bring high memory capacity and efficiency – up to 18 TB of CPU memory (LPDDR5X) is available in the rack, with exceptional bandwidth up to 14.3 TB/s aggregate. Critically, the GB300 NVL72’s design tightly couples CPUs and GPUs via 5th-generation NVLink connectivity, creating a unified memory architecture. NVIDIA NVLink provides an enormous 130 TB/s of total bandwidth between the chips, allowing GPUs to communicate with each other and with CPU memory at high speed. This means large datasets and models can be shared seamlessly across the 72 GPUs, and the CPUs can directly assist in memory-intensive tasks without bottlenecks. The platform supports a total of up to 40 TB of fast memory (GPU + CPU memory combined), making it possible to hold extremely large model parameters and context windows in memory for inference. This memory and networking integration is a breakthrough for AI at scale – GPUs can effectively act in unison on giant models that previously had to be sharded across slower network links.
Integrated High-Speed Networking (ConnectX-8 SuperNIC): Each GPU in the GB300 NVL72 is directly equipped with high-bandwidth network connectivity. NVIDIA introduced a novel ConnectX-8 “SuperNIC” I/O module that hosts dual ConnectX-8 NIC devices, providing an aggregate 800 Gb/s of network bandwidth to each GPU. In other words, every GPU can communicate out to other racks or storage at 800 Gb/s, an enormous per-node bandwidth that ensures data isn’t the bottleneck for distributed workloads. These NICs support both the latest NVIDIA Quantum-X800 InfiniBand and Spectrum-X Ethernet networks, with full RDMA (remote direct memory access) capabilities for ultra-low latency data transfers. This flexible networking means the GB300 NVL72 can be deployed in either InfiniBand-based AI supercomputers or Ethernet-based cloud data centers while still achieving “in-network computing” performance levels. The result is that multi-rack AI jobs (spanning hundreds or thousands of GPUs) can be scaled efficiently, since each GPU can send/receive data at 800 Gb/s to its peers without saturating the network. This networking prowess is especially vital for large inference pipelines and model-parallel workloads.
BlueField-3 DPUs and Secure Multi-Tenancy: In addition to CPUs and GPUs, the GB300 NVL72 platform also integrates 18 NVIDIA BlueField-3 DPUs (data processing units). These specialized accelerator cards offload networking, storage, and security tasks from the CPUs, enabling smarter and more secure data handling. With the BlueField-3 DPUs running NVIDIA’s DOCA software framework, the system can provide line-rate networking for cloud tenants up to 200 Gb/s per connection, with features like isolated virtual networks and encryption handled in hardware. This is particularly important in multi-tenant AI cloud environments – it means that even when the GB300 NVL72 is offered as a service (as on CoreWeave’s cloud), different users’ jobs can each get secure high-speed network throughput without interfering with one another. The BlueField DPUs essentially act as intelligent traffic cops and security guards for the data flowing in and out of the AI rack, all while the GPUs remain focused on crunching AI computations. By integrating DPUs, NVIDIA ensures the GB300 NVL72 is cloud-ready and enterprise-grade, with no compromises on performance or security when sharing this powerful resource.

Performance comparison: NVIDIA GB300 NVL72 vs. NVIDIA Hopper platform



Specifications of the NVIDIA GB300 NVL72

GPU Configuration: 72× NVIDIA Blackwell Ultra GPUs, fully interconnected via NVLink and NVSwitch. Each GPU provides up to ~15 PF4 (FP4 precision) peak compute (approx 15 petaflops) and contains 288 GB of HBM3e memory on board. The GPUs are arranged to work in concert as one giant accelerator, with NVLink connecting all 72 GPUs into a single high-speed cluster.
CPU Configuration: 36× NVIDIA Grace CPUs (Arm Neoverse V2 architecture). In total, the platform has 2,592 CPU cores (72 cores per Grace CPU) to handle preprocessing, data loading, and CPU-bound portions of workloads. The Grace CPUs are paired with the GPUs via NVIDIA’s NVLink-Chain and NVLink-C2C technology, ensuring tight coupling between CPU and GPU memory systems. The Grace chips also contribute significant memory capacity and are highly energy-efficient (they offer 2× the energy efficiency of traditional server CPUs for data center workloads).
Memory Capacity: GPU Memory: Up to 21 TB of HBM3e memory across the 72 GPUs (each GPU’s 288 GB adds up to ~20.7 TB total). This memory is extremely fast, with an aggregate bandwidth of ~576 TB/s (each GPU providing on the order of 8 TB/s of bandwidth from its HBM stacks). CPU Memory: Up to 18 TB of system memory (LPDDR5X ECC memory across the Grace CPUs, via SOCAMM modules) with around 14.3 TB/s total bandwidth. The platform’s architecture treats these memories in a complementary way – the GPU memory is ultra-fast for active data, while the CPU memory can hold additional data and be accessed via NVLink when needed. Combined, the GB300 NVL72 offers nearly 40 TB of addressable memory for AI workloads, enabling it to host extremely large models or datasets entirely in-memory for inference.
Interconnect (NVLink and NVSwitch): NVIDIA’s 5th-generation NVLink provides the backbone of intra-rack communication. The aggregate NVLink bandwidth is 130 TB/s across the system. In essence, all 72 GPUs are connected in a seamless mesh through NVSwitches, so any GPU can talk to any other GPU at high speed. This allows the GPUs to share data and model parameters as if they were on a single super-GPU with 21 TB of HBM. The ultra-high NVLink bandwidth is critical for scaling deep learning models across dozens of GPUs without hitting communication bottlenecks.
External Networking: As mentioned earlier, each GPU is equipped with a dual-ConnectX-8 “SuperNIC” module providing 800 Gb/s of network I/O per GPU. In terms of port configuration, this could be seen as, for example, two 400 Gb/s ports per GPU, or eight 100 Gb/s links – but from a spec perspective, 800 Gb/s per GPU is the headline. Across 72 GPUs, that’s a staggering potential bandwidth (57.6 Tb/s total) out of the rack, though in practice it will be used for redundancy and east-west clustering rather than all at once. The networking supports both Ethernet (Spectrum-X) and InfiniBand (Quantum-X800) fabrics at full rate, with RDMA and GPUDirect capabilities for direct GPU-to-GPU communication between racks or to storage. In addition, the system integrates 18 NVIDIA BlueField-3 DPUs on the networking side. While not always listed in “speeds and feeds,” these DPUs offload network processing and can provide up to 200 Gb/s of secure throughput per host/tenant connection, as well as enhanced capabilities like packet processing and storage acceleration.
Power and Cooling: Each Blackwell Ultra GPU in the GB300 NVL72 has a high TDP (Thermal Design Power) to achieve its performance – on the order of several hundreds of watts per GPU (exact figures are not publicly stated, but estimates range around 700–1000 W per GPU given the total rack power). The entire 72-GPU rack is designed for ~120 kW+ of power draw. Cooling this requires direct liquid cooling to each component. The spec is that the GB300 NVL72 rack is fully liquid-cooled, often with warm-water cooling loops that can handle >90% of the heat load. This enables deployment in AI data centers without exotic cooling infrastructure – a properly equipped facility with coolant distribution can accommodate these racks that dissipate the heat equivalent of dozens of household ovens running simultaneously. The benefit is a much higher compute density per square foot and lower cooling power overhead compared to massive air-cooling setups. (For context, the previous GB200 NVL72 rack consumed roughly 130 kW and was also liquid-cooled; the GB300 likely is in a similar or slightly higher power class, but performing 1.5× more work in that envelope).
Other Specs: The platform supports all major AI frameworks and comes with NVIDIA’s software stack (CUDA, AI Enterprise, etc.) optimized for Grace+Blackwell. It also supports Multi-Instance GPU (MIG) partitioning on the GPUs, though in inference scenarios the full GPUs are often used together. Storage is not part of the GB300 spec itself, but such a rack would be connected to high-speed storage (e.g., NVMe over Fabrics) to stream model data; the 800 Gb/s NICs and BlueField DPUs facilitate that with minimal latency.

Enhancing AI reasoning inference and throughput

First, the Blackwell Ultra GPUs in this platform come with architectural enhancements targeting inference performance. As mentioned, the Tensor Cores now support FP4 (4-bit floating point) precision and offer 2× faster attention processing than previous GPUs. The attention mechanism is critical in transformer-based models (like GPT-style LLMs) for handling long text and reasoning over it. By doubling the speed of attention calculations, the GB300 NVL72 can handle much larger context windows or more simultaneous sequences without latency blowups. Additionally, FP4 support means the GPUs can process twice as many operations in the same time (compared to 8-bit precision) when models are optimized to use 4-bit weights/activations. Many large inference models can be quantized to 4-bit with minimal accuracy loss, immediately yielding higher throughput. The Blackwell Ultra’s 1.5× increase in raw compute FLOPS (versus even baseline Blackwell GPUs) also contributes to speeding up inference computations across the board.
Second, memory capacity and throughput have been elevated to enhance inference on large models. Each GPU having 288 GB of HBM3e means that extremely large models (potentially up to tens of billions of parameters per GPU in 4-bit form) can reside in memory local to the GPU. For example, a model with 200 billion parameters in 4-bit would be ~100 GB – which fits well within 288 GB, allowing room for activations and overhead. The 1.5× larger memory per GPU (compared to the prior gen) directly translates to the ability to handle larger batch sizes and longer sequence lengths, which are key for maximizing throughput in reasoning tasks. Moreover, the high memory bandwidth (each GPU ~8 TB/s) ensures that these massive models can be fed to the compute units without stalling. The overall effect is that even the largest context AI models (such as GPT-4 style models with very long prompts) can be run at full speed, utilizing the GPUs efficiently. NVIDIA specifically notes that GB300 NVL72 boosts throughput for “the largest context lengths,” meaning scenarios like summarizing book-length texts or doing multi-step reasoning over long dialogs become much more feasible.
Third, the scale of the GPU cluster (72 GPUs with NVLink) enhances inference through parallelism. In many inference workloads, especially for huge models, you might distribute different parts of a model across multiple GPUs (model parallelism) or serve many queries in parallel (data parallelism). The GB300’s internal NVLink fabric allows these 72 GPUs to work together on a single model as if they were one unit with shared memory. This is crucial for “test-time scaling” – if a model doesn’t fit in one GPU, it can be partitioned across dozens of GPUs with minimal communication overhead, so the inference still runs efficiently. Alternatively, if you have a flood of inference requests (think of millions of users querying an AI assistant), the 72 GPUs can be carved into groups or operate independently to serve many requests concurrently. The fifth-gen NVLink and NVSwitch ensure that whichever approach is used, the GPUs remain fed with data and intermediate results without slowdowns. In essence, seamless communication between every GPU means the platform can achieve near-linear scaling in throughput as you use more GPUs for inference – a feat not easily achievable in loosely connected clusters.

Networking capabilities with the NVIDIA ConnectX-8 SuperNIC

CoreWeave: Early access deployment and partnership with NVIDIA



Conclusion

Sources

NVIDIA, “Designed for AI Reasoning Performance & Efficiency – NVIDIA GB300 NVL72,”
CoreWeave, “CoreWeave Leads the Way with First NVIDIA GB300 NVL72 Deployment,” Peter Salanki, July 3, 2025.
DataCenterDynamics, “Dell ships first Nvidia GB300 NVL72 to CoreWeave,” Charlotte Trueman, July 03, 2025.
NVIDIA Developer Blog, “NVIDIA Blackwell Architecture Highlights,” (via product page features).
CoreWeave Blog, “Why You Need Liquid Cooling for AI Performance at Scale,” Tess Sohngen, April 25, 2025.
CoreWeave Press Release, “CoreWeave and Weights & Biases Announce New Products and Capabilities, Helping AI Developers Iterate Faster on Models and Agents,” June 18, 2025.
CoreWeave Blog, “CoreWeave Leads the Way with First NVIDIA GB300 NVL72 Deployment,” (observability section).