The Missing Guide to the H100 GPU Market

🌈 Abstract

The article provides a comprehensive guide to the GPU market, particularly focusing on the Nvidia H100 GPU. It covers various aspects of GPU procurement, including pricing, reliability, hardware specifications, and location considerations. The article aims to help readers navigate the complex GPU landscape and make informed decisions about their GPU infrastructure needs.

🙋 Q&A

[01] Getting GPUs: Pricing

1. What are the current pricing trends for renting H100 GPUs?

• The most common way to access H100 GPUs is through reserved computation capacities, which provide better pricing:
  • For small-scale clusters (16-512 GPUs), the current baseline pricing is around $2.60/h for a 6-month commitment, $2.40/h for 12 months, and less than $2.20/h for longer terms.
  • For large-scale clusters (more than 512 GPUs), pricing is highly variable and depends on various factors.
• On-demand access to H100 GPUs is becoming more available, with prices around $3-$3.5 per hour across several providers, but these often come with limitations like lack of high-bandwidth GPU fabric.
• The H100 pricing is following a similar trend as the A100 GPU, with prices falling by around 20% over the past year and lead times reducing from 6 months to a few weeks or less.

2. What is the cost breakdown for buying H100 GPUs?

• Compute Hardware: $1.1/h
• Network Hardware: $0.2/h
• Power and other IDC costs: $0.3/h
• Spare parts: $0.1/h
• Total cost over 4 years: $1.7/h

3. Should I rent or buy H100 GPUs?

• The article slightly leans towards renting, citing reasons like:
  • Consistent price drops over time
  • Lower upfront cost
  • Better flexibility to scale up or down

[02] Using GPUs: Reliability

1. What are the key considerations for ensuring GPU reliability?

• Thorough pre-production and burn-in testing before delivery to eliminate issues
• Extensive active monitoring during operation to detect early signs of failures
• Availability of onsite staffing support and SLA guarantees from the provider

2. How does Lepton approach GPU reliability?

• Lepton conducts comprehensive testing, including running popular training frameworks, to verify end-to-end performance before delivery.
• Lepton deploys the open-source tool GPUd on every machine to actively monitor GPU health and related components, enabling proactive maintenance and minimizing disruptions.
• Lepton's onsite staff can quickly identify and resolve issues, backed by SLA guarantees.

[03] Hardware Specs: a datacenter view

1. What are the key hardware considerations beyond the GPUs themselves?

• GPU Servers: Vendor-specific factors like PSU, cooling, and hardware-related issues
• GPU Network: Choice between InfiniBand and RoCE, with RoCE being a viable and more cost-effective option for smaller clusters
• CPU/Memory: Fully utilizing the available CPU and memory resources to avoid bottlenecks
• Storage: Minimum 20TB of local NVMe storage, with options for remote storage solutions like NFS or object storage

2. How does location impact GPU infrastructure?

• North America is the most popular choice due to lower power costs, affordable network rates, and better availability of parts.
• Europe ranks second, while Asia Pacific generally has higher costs.
• For training, location is less critical, but for inference, latency and reliability are significant factors, so it's important to position the infrastructure close to the customer base.