A 35B MoE on a 16 GB GPU, without the offload tax

A 33-35B MoE only fires ~8 of 256 experts per token, but holding it on the GPU costs you all of them. Luce Spark pins the experts your traffic uses, offloads the rest to CPU, and decodes the whole token in one fused graph, so a 33-35B MoE fits a 16 GB card and still runs near full-GPU speed (~100 tok/s vs ~119 all-GPU on a 3090). Self-tuning, one flag.
7 points by GreenGames 136 days ago www.lucebox.com 0 comments
Summary not yet generated for this article.

Comments (0 of ) ⌛

Loading more comments... ⌛
Scroll down to load more comments
No comments available.
Type to search HN and engineering blog articles
Press Escape to close  |  ↑↓ to navigate  |  Enter to open