Show HN: A minimalist proxy for your local LLM cluster (~1100 lines)
Lightweight API key proxy for llama.cpp servers with per-user token usage tracking. - robolamp/smol-llm-proxy
Summary not yet generated for this article.
Original Text
Hi all!
I made this small proxy when I built a small GPU cluster at home and wanted to share it with friends while keeping speed limits and token accounting.
I also put a lot of effort in making it work as quick as possible, and IMO the result is worth sharing!
It is not trying to be LiteLLM: in fact, I wanted to make it opposite, as small as possible, with a small amount of dependencies and without cloud connections by default.
Will be happy to hear your feedback :)
PS also installable with pip install smol-llm-proxy
Comments (0 of ) ⌛