Categories

Natural Language Processing

Libraries for working with human languages.

Repos

81

Stars

34,035

30 days

+65

Morphological Analyzers

ikawaha

Self-contained Japanese Morphological Analyzer written in pure Go

Created Jun 2014

hacktoberfest japanese japanese-language korean morphological-analysis
Total 989
30 days +4

Tokenizers

neurosnap

A multilingual command line sentence tokenizer in Golang

Created Aug 2015

cli sentence-tokenizer sentences tokenizer
Total 472
30 days +1

More

alasdairforsythe

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

Created May 2023

text-tokenization tokenisation tokenization tokenize tokenizer
Total 626
30 days +2
sugarme

NLP tokenizers written in Go language

Created Aug 2020

deep-learning golang-tokenizer nlp tokenizer
Total 342
30 days +1
clipperhouse

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.

Created Apr 2020

go golang nlp tokenization tokenizer
Total 125
30 days +1
clipperhouse

Tokenizers and lemmatizers for Go

Created May 2018

data-science go lemmatizer nlp tokenizer
Total 120
30 days +0
samber

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

Created Dec 2022

bpe byte-pair-encoding codex decoder encoder
Total 82
30 days +0

From awesome-go (MIT), plus repos sorted by Jev.