Multi-Token Prediction Done Right
3 points by matt_d
Multi-Token Prediction (MTP) enhances language model pre-training by improving performance and speeding up inference. However, a flaw in the standard MTP training process causes interference between next-token and multi-token learning, particularly as model size increases. The study reveals that the MTP cross-entropy gradient can be broken down into a KL self-distillation term that negatively impacts the learning of each MTP head.