Categories / Natural Language Processing

tiktoken-go

tiktoken-go / tokenizer

Pure Go implementation of OpenAI's tiktoken tokenizer

Last push 4 weeks ago

Created

Apr 2023

Total stars

463

7 days

0

30 days

+5

12 months

+64

Star trends

README badge

Best of Go badge for tiktoken-go/tokenizer
[![Best of Go](https://bestofgo.dev/badge/tiktoken-go/tokenizer.svg)](https://bestofgo.dev/tiktoken-go/tokenizer)