feat: TQ4_1S weight compression (Metal only, needs CUDA port) by TheTom · Pull Request #45 · TheTom/llama-cpp-turboquant

TurboQuant model weight compression support added to Llamacpp

Summary

TQ3_1S (3-bit, 4.0 BPW) and TQ4_1S (4-bit, 5.0 BPW) weight quantization using WHT rotation + Lloyd-Max centroids
V2.1 fused Metal kernel: zero threadgroup memory, cooperative SIMD rotation...

10 points by lastdong 201 days ago github.com 3 comments

Summary not yet generated for this article.

Comments (3 of ) ⌛

Loading more comments... ⌛

Scroll down to load more comments

No comments available.