Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP
How I built a 5.01 BPW Qwen3.8 27B hybrid, filled its 256K context, tuned MTP and gained 21.97% with a measured llama.cpp runtime.
Summary
A detailed account of building a high-performance Qwen3.8 27B model with a focus on optimizing its context and tuning for maximum throughput. The author shares specific metrics and results, making it a valuable read for those interested in advanced AI model configurations.
Comments (26 of ) ⌛