Why Is Speculative Decoding Fast?
It's not because you do less work. It's because you change the kind of work the GPU is doing.
Summary
Speculative decoding enhances GPU performance by altering the type of tasks it handles, rather than reducing the workload. This shift in approach is key to understanding its speed advantages.
Comments (5 of ) ⌛