![[Tech] Why MLA and MTP Fight Each Other: Attention Through Arithmetic Intensity](https://changyi.fun/images/attention-ai-og.png)
Attention Through Arithmetic Intensity
Reduce MHA to its arithmetic intensity and you get exactly 1. GQA, MQA and MLA come out just as clean -- head counts and nothing else. That last constant is what puts MLA at odds with MTP.
Summary
The article breaks down the arithmetic intensity of various models, revealing that MHA consistently equals 1 while comparing GQA, MQA, and MLA. It also highlights a key difference between MLA and MTP, making it a valuable read for those interested in model efficiency.
Comments (0 of ) ⌛