Width vs. Depth: Speculating on the Margin
Some thinking about how to trade off batching and speculative decoding in a running inference engine.
Summary not yet generated for this article.
Comments (1 of ) ⌛