LLM inference providers

A new write-up from Unblocked describes adaptive routing of large language model requests across multiple inference providers using an approach modelled on TCP congestion control. The system treats providers like network paths, backing off when a provider slows down and shifting traffic toward faster responses. The approach is drawing attention among developers interested in reliability and latency for LLM applications.

Discussion is growing around the business economics of offering large language model inference as a cloud service. Commenters and industry observers are examining why serving AI models to users is so costly, how providers price access, and whether current pricing models are sustainable given the compute demands. The debate touches on GPU costs, margins, and competition among AI service providers.