Categories / Natural Language Processing
alasdairforsythe / tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
Last push 4 months ago
text-tokenization
tokenisation
tokenization
tokenize
tokenizer
tokenizing
vocabulary
vocabulary-builder
vocabulary-generator
Created
May 2023
Total stars
625
Last 7 days
+1
Last 30 days
+1
Last 12 months
+27
README badge
[](https://bestofgo.dev/alasdairforsythe/tokenmonster)