GitHub topics

tokenizer

Mostly in Natural Language Processing 58%

Repositories

12

Total stars

3,588

Trending

mattn

A tiny neural-network framework in pure Go with AVX2 SIMD kernels (GOEXPERIMENT=simd)

Created Aug 2026

autograd avx2 deep-learning gguf golang
Total 120
30 days +5
ikawaha

Self-contained Japanese Morphological Analyzer written in pure Go

Created Jun 2014

hacktoberfest japanese japanese-language korean morphological-analysis
Total 989
30 days +4
alasdairforsythe

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

Created May 2023

text-tokenization tokenisation tokenization tokenize tokenizer
Total 626
30 days +2
bzick

Tokenizer (lexer) for anything

Created Oct 2021

golang lexer parse parser tokenizer
Total 143
30 days +1
sugarme

NLP tokenizers written in Go language

Created Aug 2020

deep-learning golang-tokenizer nlp tokenizer
Total 342
30 days +1
clipperhouse

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.

Created Apr 2020

go golang nlp tokenization tokenizer
Total 125
30 days +1
neurosnap

A multilingual command line sentence tokenizer in Golang

Created Aug 2015

cli sentence-tokenizer sentences tokenizer
Total 472
30 days +1
timtadh

Lex machinary for go.

Created Apr 2014

dfa go lex lexer lexical-analysis-engines
Total 412
30 days +1

Top

ikawaha

Self-contained Japanese Morphological Analyzer written in pure Go

Created Jun 2014

hacktoberfest japanese japanese-language korean morphological-analysis
Total 989
alasdairforsythe

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

Created May 2023

text-tokenization tokenisation tokenization tokenize tokenizer
Total 626
neurosnap

A multilingual command line sentence tokenizer in Golang

Created Aug 2015

cli sentence-tokenizer sentences tokenizer
Total 472
timtadh

Lex machinary for go.

Created Apr 2014

dfa go lex lexer lexical-analysis-engines
Total 412
sugarme

NLP tokenizers written in Go language

Created Aug 2020

deep-learning golang-tokenizer nlp tokenizer
Total 342
bzick

Tokenizer (lexer) for anything

Created Oct 2021

golang lexer parse parser tokenizer
Total 143
clipperhouse

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.

Created Apr 2020

go golang nlp tokenization tokenizer
Total 125
mattn

A tiny neural-network framework in pure Go with AVX2 SIMD kernels (GOEXPERIMENT=simd)

Created Aug 2026

autograd avx2 deep-learning gguf golang
Total 120
clipperhouse

Tokenizers and lemmatizers for Go

Created May 2018

data-science go lemmatizer nlp tokenizer
Total 120
openshieldai

OpenShield is a new generation security layer for AI models

Created Jun 2024

ai artificial-intelligence firewall golang guardian
Total 87
samber

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

Created Dec 2022

bpe byte-pair-encoding codex decoder encoder
Total 82
soulteary

Self-hosted browser tool for counting and inspecting text tokens, with a Chinese UI adapted from the OpenAI Tokenizer.

Created Dec 2023

docker golang openai openai-tokenizer self-hosted
Total 70

Related

All topics →