Categories / Natural Language Processing

alasdairforsythe

alasdairforsythe / tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

Last push 4 months ago

text-tokenization tokenisation tokenization tokenize tokenizer tokenizing vocabulary vocabulary-builder vocabulary-generator

Created

May 2023

Total stars

626

7 days

+1

30 days

+2

12 months

+28

Star trends

README badge

Best of Go badge for alasdairforsythe/tokenmonster
[![Best of Go](https://bestofgo.dev/badge/alasdairforsythe/tokenmonster.svg)](https://bestofgo.dev/alasdairforsythe/tokenmonster)