Abstract:

With the explosive growth in demand for large models inference，traditional centralized or static multi-data center deployment models face severe challenges in latency，data compliance，and resource elasticity. This paper proposes a cloud-edge collaborative wide-area distributed inference network architecture，focusing on building a new intelligent-computing service system for the emerging computing-power internet. The architecture introduces a prefill-decode separation mechanism： the latency-sensitive prefill stage is offloaded to edge nodes closer to data sources，while the high-throughput decode stage is deployed in the central cloud，enabling secure collaboration over a wide-area network.

Key words: wide-area distributed inference, prefill-decode separation, large model inference