LLM inference providers

A new engineering write-up describes a method for routing large language model requests across multiple inference providers using congestion-control ideas borrowed from TCP. The approach adaptively shifts traffic toward providers with lower latency or higher throughput, easing bottlenecks when one provider slows down. Commenters are discussing the trade-offs of applying classic networking techniques to AI serving infrastructure.