Alibaba Unveils AI System to Slash GPU Costs by Over 80%
Alibaba Group Holding Ltd. has developed a new AI computing solution that dramatically reduces the number of expensive GPUs required to run large language models, a breakthrough that could significantly lower the high costs a of generative AI services.
The solution, named "Aegaeon," was developed by Alibaba Cloud, the company's cloud computing arm. Its core technology, which addresses a critical issue of resource waste in AI model deployment, has already been integrated into Alibaba’s Bailian AI platform.
The company's research paper on Aegaeon has been accepted by SOSP 2025, a top-tier academic conference for computer systems research. This acceptance underscores the significance of the software-level innovations aimed at optimizing AI infrastructure.
In a three-month beta test, Aegaeon cut the number of Nvidia H20 GPUs needed to serve dozens of large models—some with up to 72 billion parameters—from 1,192 to just 213. This represents an 82% reduction in required hardware, pointing to substantial capital expenditure savings for companies deploying AI at scale.
Tackling Widespread GPU Inefficiency
Aegaeon is designed to solve a fundamental problem in AI services: the underutilization of graphics processing units. In typical deployment scenarios, a small number of popular models like Alibaba's Qwen handle the vast majority of user requests. However, a large number of less-frequently used "long-tail" models remain idle while occupying dedicated GPU resources.
This creates significant inefficiency. Data from Alibaba’s own model marketplace showed that 17.7% of its GPU computing power was being used to process just 1.35% of total requests. Aegaeon aims to dismantle this inefficient "one model, one GPU" paradigm through a resource pooling system.
Token-Level Scheduling at the Core
The system's core innovation is a technique called "token-level scheduling." Instead of dedicating a GPU to a single model, Aegaeon’s service system can dynamically decide whether to switch to a different model after each token is generated. This allows for extremely fine-grained management of computing resources.
To ensure this process is fast enough for real-time applications, the system employs a suite of optimizations, including component reuse, refined memory management, and synchronization of the KV cache. These improvements have reduced the overhead costs associated with switching between models by 97%, enabling model-switching responses at a sub-second level.
Boosting Throughput, Cutting Costs
The performance gains from Aegaeon are substantial. According to Alibaba, the system allows a single GPU to serve as many as seven different models simultaneously. This translates to a 1.5 to 9-fold increase in effective throughput and a 2 to 2.5-fold increase in request-processing capacity compared to current mainstream solutions.
As the global AI industry continues to expand, the focus is shifting beyond simply adding more hardware. Software-level innovation that can extract more performance from existing hardware is becoming a key competitive advantage. By optimizing system software, companies can unlock greater efficiency and make the deployment of advanced AI more economically viable, a development with direct implications for a sector heavily reliant on high-cost hardware.