A simple fix for LLM tail latency
Send every request twice and take the faster response.
Summary
A straightforward approach to reducing latency in large language models involves sending each request twice and selecting the quicker reply. This method could significantly enhance response times in various applications.
Comments (16 of ) ⌛