Show HN: KV-psi, using Linux PSI to to trim an LLM KV cache
Contribute to infiniteregrets/kv-psi development by creating an account on GitHub.
Summary not yet generated for this article.
Original Text
I thought it'd be interesting to use Linux PSI (Pressure Stall Information) for an LLM runtime to trim the KV cache. This is mainly useful imo for edge devices like the Jetson Orin super nano kit which have unified memory. I haven't benched much, but plan to do so more over time and see if I can make a real use of it as I run local LLMs. Let me know if it makes sense :P (I of course vibed this idea)
Comments (0 of ) ⌛