Microsoft Announces Maia AI, Arm CPU, AMD MI300 and New Nvidia for Azure

If you missed Satya Nadella’s keynote speech, here’s a summary of the huge range of news and my perspective. Big impact on MSFT, AMD, AWS and NVDA.

CEO Satya Nadella presented a keynote speech full of new product announcements. If you haven’t seen it, here’s the news in brief, followed by my perspective.

1. New hollow fiber optical cable for low-energy, high-performance networking
2. New Microsoft Maia AI Accelerator: Better than AWS, but less HBM memory than NVIDIA and AMD for large AI model training and inference
3. New AMD MI300 instances for Azure: a serious challenge to NVIDIA H100
4. New NVIDIA H200 Instances Coming: More HBM Memory
5. New NVIDIA AI Foundry services (and Jensen on stage!)
6. New (okay, that word is getting old here 😉 Microsoft Cobalt Arm CPU for Azure: 40% faster than the existing Ampere Ultra

Description and viewpoint

Maiya 100

First of all, there is no rumored chiplet cloud architecture for inference processing. But it’s a good start for Microsoft, if for no other reason than to give Microsoft more pricing leverage with NVIDIA and AMD. And it has enough performance for many internal workloads. MSFT has stepped up its semiconductor game significantly. Although not leading overall, they are now competitive, especially with Amazon, which appears to be far behind in performance.

Maia is built on TSMC 5nm, and has strong TOPS and FLOPS, but it was designed before the LLM explosion (an ASIC takes ~3 years to develop, fab, and test). It’s huge with 105B transistors (vs 80B in the H100). It cranks out 1600 TFLOPS of MXInt8 and 3200 TFLOPS of MXFP4. Its biggest drawback is that it only has 64GB of HBM but a ton of SRAM. In other words. It seems to be designed for older AI models like CNN. Microsoft went with only four stacks of HBM instead of 6 like Nvidia and 8 like AMD. The second generation memory bandwidth is 1.6 TB/s, which beats AWS Tranium/Inferentia at 820 GB/s and is significantly less than NVIDIA, which has 2×3.9 TB/s.

AI performance will depend on what you are doing. For LLM training, both NVIDIA and AMD should make it easy to complete. Inference for larger models will also perform much better on GPUs, but for smaller models like enterprises this should be quite enough.

cobalt 100

The Cobalt 100 CPU follows and probably for all practical purposes replaces the Ampere Arm CPU in Azure. It brings 128 Neoverse N2 cores and 12 channels of DDR5 on Armv9. The Arm Neoverse N2 offers 40% more performance than the Neoverse N1. Microsoft built the Cobalt 100 using Arm’s Neoverse Genesis CSS (Compute Subsystem) platform.

NVIDIA Foundation Services and H200 on Azure

Microsoft and NVIDIA have been partnering for years. Unlike many other cloud service providers, Azure supports all of NVIDIA’s technologies without any changes, adopting the leading hardware, networking, and software that has given NVIDIA market leadership.

Newly announced NVIDIA Foundry Services on Azure provides an end-to-end collection of NVIDIA AI Foundation models, NVIDIA NeMo framework and tools, NVIDIA AI Enterprise, and NVIDIA DGX cloud AI supercomputing – enabling startups and enterprises to build custom AI models and Available to deploy. On Microsoft Azure. Additionally, Microsoft announced the new NVIDIA H200 GPU as a service, available in 2Q 2024, and TensorRT-LLM on Windows.

AMD MI300 Preview on Azure

As expected, Microsoft has also decided to bring the upcoming AMD MI300 Instinct GPU to Azure, which could become a major alternative to NVIDIA GPUs for customers. This is an important step for both Azure and AMD, as many developers want to get their hands on this new technology, which will help accelerate the necessary software development on the MI300. Thanks to Microsoft, everyone will be rushing to test it, and we’ll know a lot more, a lot faster.

We believe the MI300 may be best suited for enterprises looking to reduce inference costs, but will also perform well for training. We expect AMD to include some benchmarks in its December launch.

conclusion

Microsoft is adopting Switzerland’s approach to AI acceleration, supporting its Maia 100 platform for internal use (as it does for OpenAI), and NVIDIA’s and now AMD’s industry-leading platforms for cloud services. As it does today with Llama2 on the software front. From meta. Azure supports Intel, AMD, and Arm for CPUs, providing customers with a wide range of choices to meet their specific performance and cost requirements.

We believe this approach clearly differentiates Microsoft from its CSP competitors.