Repositories
12
Total stars
3,588
Trending
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
Created May 2023
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
Created Apr 2020
Lex machinary for go.
Created Apr 2014
Top
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
Created May 2023
Lex machinary for go.
Created Apr 2014
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
Created Apr 2020
Tokenizers and lemmatizers for Go
Created May 2018
OpenShield is a new generation security layer for AI models
Created Jun 2024
Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3
Created Dec 2022
Self-hosted browser tool for counting and inspecting text tokens, with a Chinese UI adapted from the OpenAI Tokenizer.
Created Dec 2023