A study of sequence weighting at scale
TL;DR: We study the scaling laws of data weighting across in-house and open-weight LMs, finding non-monotonic behavior across scales. We vary the weight assi...
Summary
Research reveals unexpected patterns in data weighting for language models, highlighting non-linear behaviors that could influence future model training strategies. Understanding these scaling laws may lead to more effective use of both in-house and open-weight systems.
Comments (1 of ) ⌛