Show HN: Makes local LLMs faster and more reliable by optimizing for your device
Frees 300+ MB of RAM per request. Cuts first-word latency by up to 53%. Drop-in for Ollama.
Summary not yet generated for this article.
Original Text
Time to first token is 39% faster
Agent wall times decrease by 46%
No swaps
Tracks your resource usage in real-time and adjusts how the model runs so that it works perfectly on your device.
Implements KV cache sizing, prefix caching, live RAM pressure management, context trimming, KV quantization, and more.
Built a ton of features
Comments (0 of ) ⌛