Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine. - sqliteai/waste
Summary
The Kimi K3 model, with 2.78 trillion parameters, can now be run using just 29 GB of RAM by streaming weights directly from NVMe storage. This approach utilizes a lightweight C inference engine without dependencies, making it easier to integrate into various applications.
Comments (163 of ) ⌛