GitHub - marcelroed/gigatoken: Language model tokenization at GB/s

GigaToken: ~1000x faster Language model tokenization

Language model tokenization at GB/s. Contribute to marcelroed/gigatoken development by creating an account on GitHub.
610 points by syrusakbary 91 days ago github.com 119 comments
Summary not yet generated for this article.

Comments (119 of ) ⌛

Loading more comments... ⌛
Scroll down to load more comments
No comments available.
Type to search HN and engineering blog articles
Press Escape to close  |  ↑↓ to navigate  |  Enter to open