Show HN: I built a tool showing how AI providers (should) throttle their models

Large language model providers are compute constrained, and their universal response to congestion is to degrade service: route queries to smaller models, cut reasoning effort, truncate context. The industry's accounting says this saves money. We show the accounting is wrong, because it prices a query when the customer buys an answer. A degraded answer fails with some probability, and a failed answer either returns as a retry, inflating arrivals when the system is most loaded, or departs as churn, destroying lifetime value on a ledger no cost dashboard displays. We model inference allocation with three classical primitives: a newsvendor whose stockout cost is churned lifetime value, a geometric retry multiplier in which the recycled product is dissatisfaction, and a two-regime transient queue whose arrival rate is made endogenous by retries. Statically, there is a nonempty, measurable regime in which a cheaper model saves energy per satisfied answer while consuming strictly more capacity per satisfied answer, so the discount inverts exactly when capacity binds. Dynamically, a reactive throttle fired during a surge can cross an ignition threshold beyond which it manufactures more traffic than it sheds, and a release rule set below the degraded equilibrium converts a transient surge into a permanent degraded regime. With heterogeneous customers, throttling is a transportation problem in retry-inflated load whose optimal policy rations intelligence by critical ratio, class by class, and whose dual, the shadow price of intelligence, prices a marginal query by class and by hour; closed-form trajectories make it computable in milliseconds. Stochastic analysis sharpens rather than erodes the thesis: the ignition boundary acquires a predicted width, and noise punishes the reactive policy that parks the system against it. Under congestion, throttling is not a cost lever but a demand lever.

3 points by eliotho 57 days ago arxiv.org 0 comments

Summary not yet generated for this article.

Original Text

OP here: this project was born out of the frustration/paranoia that AI providers are throttling their models when their server load is too high. So, I set out to model and study the problem mathematically to understand what was happening, what I found was quite surprising.

The idea seems natural: as the data center demand increases momentarily through the day, throttling their models (either using a quantized version, reducing the context window or lowering the tier of the model to a smaller one) seems appealing as the model in principle uses less electricity. The problem is that this can cause in fact the opposite effect: as users are trying to solve a question, if the AI gives a bad answer, the user is likely to keep re-asking. On the AI provider side this looks paradoxical: throttling to a lower model creates in fact more demand for their data center.

This problem is even worse for agentic workflows, as these are more likely to create a re-ask storm, and maybe explains the outages and anecdotal experiences of users that feel the models are degraded.

The model: I used mainly queueing theory arguments solving the optimal scheduling serving for an AI fleet with heterogeneous users.

  Insights: The industry standard practice of throttling once the number of users in system is in fact what’s causing the problem, the optimal rule implies separating users that won’t feel degradation as much with users that are very sensitive to it.

Limitations: The visualization and paper examples are a toy example to illustrate the problem, only the providers have enough data to properly calibrate these instances.

  Technical Details: The visualization is around 100 lines of flask plus js frontend (LLM assisted with ground truth based on the original numerical example of the paper).

Original Text

Comments (0 of ) ⌛