Janas: Run LLMs larger than RAM by streaming experts from SSD
37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than y...
Summary
A new method allows large language models to be run on hardware with less RAM by streaming data from SSDs, resulting in significantly faster processing speeds compared to existing solutions like llama.cpp. This could enable more efficient use of resources for AI tasks on consumer-grade machines.
Comments (77 of ) ⌛