Repos
81
Stars
34,035
Last 30 days
+65
Morphological Analyzers
Tokenizers
More
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
Created May 2023
text-tokenization
tokenisation
tokenization
tokenize
tokenizer
Total
626
30 days
+2
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
Created Apr 2020
go
golang
nlp
tokenization
tokenizer
Total
125
30 days
+1
Tokenizers and lemmatizers for Go
Created May 2018
data-science
go
lemmatizer
nlp
tokenizer
Total
120
30 days
+0
Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3
Created Dec 2022
bpe
byte-pair-encoding
codex
decoder
encoder
Total
82
30 days
+0
From awesome-go (MIT), plus repos sorted by Jev.