Categories

LLM Infrastructure

Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.

Repos

200

Stars

636,933

30 days

+26,248

ollama

Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

Created Jun 2023

deepseek gemma glm go golang
Total 187,904
30 days +4,126
The-Vibe-Company

An open-source engine that turns continuous content streams into search and monitoring. Durable ingestion, hybrid search, alerts, and plugins for formats, models and business rules.

Created May 2023

ai api chatbot chatgpt database
Total 39,691
30 days +123
Tencent

Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.

Created Jul 2025

agent agentic ai chatbot dsh-plugin
Total 37,868
30 days +3,034
tbphp

Self-hosted AI gateway for multi-channel, multi-credential setups — API keys and subscription accounts, scheduling, failover, request logs and usage. 自托管 AI 网关:多渠道多凭据统一接入,含密钥与订阅账号、调度容错、日志与用量。

Created Jun 2025

ai-gateway anthropic api-gateway claude claude-code
Total 7,233
30 days +105
vllm-project

An open, programmable decision layer for models and compute.

Created Aug 2025

ai-gateway guardrails inference kubernetes llm
Total 6,518
30 days +285
mostlygeek

Reliable model swapping for any local AI server. Chosen by thousands of users. Batteries included. If we don't have it, you probably don't need it.

Created Oct 2024

ai golang llama llm localllama
Total 6,284
30 days +255
looplj

⚡️ Open-source AI Gateway — Use any SDK to call 100+ LLMs. Built-in failover, load balancing, cost control & end-to-end tracing.

Created Sep 2025

agent agents ai anthropic anthropic-api
Total 5,487
30 days +129
theagentrouter

Manages Unified Access to Generative AI Services built on Envoy Gateway

Created Oct 2024

ai ai-gateway api-gateway envoy envoyproxy
Total 2,430
30 days +175
beam-cloud

Ultrafast serverless GPU inference, sandboxes, and background jobs

Created Nov 2023

autoscaler cloudrun cuda developer-productivity distributed-computing
Total 1,860
30 days +32
bricks-cloud

🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, vLLM, and open-source LLMs.

Created Jul 2023

ai anthropic api artificial-intelligence azure
Total 1,234
30 days +6
kittors

Self-hosted AI gateway for coding CLIs — one OpenAI/Claude/Gemini/Codex-compatible endpoint, with a multi-tenant web console, request logs, and spend quotas.

Created Feb 2026

ai-gateway anthropic api-gateway claude-code gemini-cli
Total 1,061
30 days +27
ntthanh2603

✨Reverse-engineered API for Gemini web app. It can be used as a genuine API key from OpenAI, Gemini, and Claude.

Created Dec 2025

ai api chatbot claude docker
Total 676
30 days +78
kaito-project

🏗️ Fine-tune, build, and deploy open-source LLMs easily!

Created Sep 2023

ai buildkit chatgpt docker fine-tuning
Total 544
30 days +6
WizisCool

Modern web console for CLIProxyAPI v8+ with provider and OAuth management, live routing, request analytics, pricing, and MCP.

Created Sep 2026

ai-gateway cliproxyapi cpa dashboard golang
Total 466
30 days +252
messkan

Cut LLM costs by up to 80% and unlock sub-millisecond responses with intelligent semantic caching.A drop-in, provider-agnostic LLM proxy written in Go with sub-millisecond response

Created Nov 2025

ai badgerdb cache claude cost-optimization
Total 427
30 days +4
matrixhub-ai

An Open-source, self-hosted AI model hub with Hugging Face compatibility, accelerating vLLM/SGLang performance.

Created Jan 2026

artificial-intelligence dynamo huggingface kubernetes llm
Total 366
30 days +11
wangle201210

基于eino+gf+vue实现知识库的rag

Created Apr 2025

chat eino es go goframe
Total 328
30 days +3
sgl-project

A workload for deploying LLM inference services on Kubernetes

Created Aug 2025

k8s llm pd-disagg sglang
Total 316
30 days +9
InftyAI

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

Created Nov 2023

huggingface inference inference-platform kubernetes llamacpp
Total 315
30 days +1
luqman-v1

High-performance Go proxy gateway for 9Router LLM routing — 32K RPS, 42MB RAM, CGO-free

Created Jul 2026

9router api-gateway claude deepseek gpt
Total 304
30 days +95
T-Crypt

Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.

Created Oct 2026

comfyui golang gpu inference llama-cpp
Total 300
30 days +186
raketenkater

llama.cpp/ik_llama.cpp launcher: loads big MoE models across mismatched multi-GPU rigs by exact VRAM math.

Created Mar 2026

cuda gguf golang inference-server llama-cpp
Total 284
30 days +3
tensorchord

Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)

Created Jul 2023

cluster-manager hacktoberfest inference llm llmops
Total 282
30 days +0
teilomillet

A lightweight, production-ready RAG (Retrieval Augmented Generation) library in Go.

Created Jul 2024

ai chromadb document-search embeddings golang
Total 228
30 days +2
vibe-coding-labs

JoyCode API 代理 — 协议翻译成 Anthropic/OpenAI 兼容格式,让 Claude Code 直接调用 JoyCode支持的模型,比如GLM 5.1等

Created Apr 2026

anthropic-api api-gateway claude-code fastapi llm
Total 218
30 days +4
inference-gateway

An open-source, cloud-native, high-performance gateway unifying multiple LLM providers, from local solutions like Ollama to major cloud providers such as OpenAI, Groq, Cohere, Anthropic, Cloudflare and DeepSeek.

Created Jan 2025

agnostic anthropic api deepseek-v4-pro gateway
Total 208
30 days +7
lordmathis

Unified management and routing for llama.cpp, MLX and vLLM models with web dashboard.

Created Jul 2025

llama-cpp llama-server llamacpp llm llm-inference
Total 162
30 days +2
EinStack

🐦 A open blazing-fast simple model gateway for rapid development of production GenAI apps

Created Dec 2023

ai gateway gateway-api genai generative-ai
Total 160
30 days +0
azrinaldi

Browserless Chrome uTLS + HTTP/2 OpenAI-compatible API daemon for Google Gemini Web (/v1/chat/completions & /v1/responses)

Created Oct 2026

bridge gemini gemini-api golang http2
Total 152
30 days +58
voidmind-io

Privacy-first LLM proxy and AI gateway - load balancing, multi-provider routing, API key management, usage tracking, rate limiting. Self-hosted. Zero knowledge of your prompts.

Created Mar 2026

ai ai-gateway anthropic api-gateway docker
Total 149
30 days +5
regolo-ai

brick is a smart AI Models router, based on complexity & capabilities extraction from the query to the models via proprietary spatial embedding algorythm

Created Feb 2026

claude-code codex llm open-models routing
Total 133
30 days +4
Zhengyuuuui

将 Qoder 的 AI 服务转化为兼容 OpenAI/Claude/Codex 格式的 API,支持每日签到领 Credits、多账号管理、模型上下文展示

Created Jul 2026

api-gateway claude golang llm openai
Total 125
30 days +43
kubeagi

A diverse, simple, and secure all-in-one LLMOps platform

Created Jul 2023

agents golang kubernetes langchain large-language-models
Total 112
30 days +0
gibram-io

GibRAM is an in-memory knowledge graph server designed for retrieval augmented generation (RAG / GraphRAG) workflows.

Created Jan 2026

graph graph-database graphrag llm llms
Total 107
30 days +0
raiyanyahya

Selfhost modern LLM stacks. Run the whole fleet from your terminal

Created Jun 2026

ai-agents bubbletea cli docker fastapi
Total 105
30 days +0
gitctrlx

Llama from scratch in Go.

Created Aug 2025

llama llm llm-inference
Total 105
30 days +0
llmariner

Extensible generative AI platform on Kubernetes with OpenAI-compatible APIs.

Created Mar 2024

ai autoscaling fine-tuning gpu inference
Total 97
30 days +0
aqstack

mimir is a drop-in proxy that caches LLM API responses using semantic similarity, reducing costs and latency for repeated or similar queries.

Created Dec 2025

caching cost-optimization golang kubernetes llm
Total 94
30 days +0
ShadowSafin

AI inference gateway and control plane for multi-provider routing, policies, caching, tools, and observability.

Created Oct 2026

ai ai-gateway api-gateway inference llm
Total 93
30 days +3
greynewell

Route inference across providers.

Created Sep 2024

ai-gateway ai-infrastructure anthropic api-gateway cost-tracking
Total 93
30 days +0
hypit-ai

Multi-tenant LLM API gateway for Claude and Codex — single Go binary, USD wallet, Alipay top-up, per-token caps, status-page health, fumadocs-style /docs.

Created May 2026

alipay anthropic api-gateway claude codex
Total 84
30 days +25
adrianliechti

Inference Hub for AI at Scale

Created Sep 2023

ai-gateway azure-openai claude function-calling groq
Total 82
30 days +1
kodelyx

🚀 Free Unlimited OpenAI-compatible API for Google Gemini 4 Hydrogen & Neon. Supports Deep Thinking controls (High/Medium/Low), Agentic Task Mode, HTTP/3 QUIC, and multi-account pooling.

Created Feb 2026

ai-agents chrome-extension cursor deep-thinking docker
Total 78
30 days +62
rasonyang

Self-hosted voice gateway in Go that speaks the OpenAI Realtime (GA) WebSocket protocol and runs a cascaded ASR → LLM → TTS pipeline behind it. Keep clients unchanged while swapping speech and LLM providers at runtime

Created Sep 2026

asr call-center callcenter cascaded-pipeline dashscope
Total 65
30 days +5
PIXEL-API

基于 Sub2API 二开的 AI API 网关:多平台账号接入、账号共享与广场调度、Token 级计费与分账

Created May 2026

account-sharing ai-gateway api-gateway claude gemini
Total 65
30 days +1
drbh

💬 modular open source LLM server, web and desktop app

Created May 2023

ai chatbot containers gpt llm
Total 63
30 days +0
avilum

A client/server for LLaMA (Large Language Model Meta AI) that can run ANYWHERE.

Created Mar 2023

ai client-server facebook llama llm
Total 62
30 days +0
ikhsan3adi

Convert Google Gemini web into OpenAI-compatible API. Zero cost, single static binary.

Created Aug 2026

9router ai-agent ai-proxy free-ai free-token
Total 61
30 days +13
jitllm

Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory

Created Sep 2026

cuda inference inference-acceleration inference-engine inference-server
Total 58
30 days +7
hugalafutro

Multi-Provider AI Gateway, "Because we have LiteLLM at home" No personal logs by design. Model autodiscovery, Failover groups, High availability, Android companion app, and more!

Created Apr 2026

ai ai-tools android-app anthropic-api discovery
Total 58
30 days +0
ZiChuanLan

OpenAI-compatible multi-channel LLM relay gateway with admin console

Created Jul 2026

ai-gateway anthropic api-gateway gemini golang
Total 55
30 days +14
canonical

Local inference, optimized for your hardware

Created Jun 2025

ai inference llm local-ai model-serving
Total 55
30 days +2
localcloud-sh

Stop paying for AI APIs during development. LocalCloud runs everything locally - GPT-level models, databases, all free.

Created Jun 2025

ai ai-app ai-development ai-development-tools ai-infrastructure
Total 51
30 days +0
scitix

Arks is a cloud-native inference framework running on Kubernetes

Created Apr 2025

ai cloudnative-services dynamo inference kubernetes
Total 51
30 days +0
MegaGrindStone

A Go library implementation of LightRAG - an advanced Retrieval-Augmented Generation (RAG) system that uniquely combines vector databases with graph database relationships to enhance knowledge retrieval.

Created Mar 2025 Archived

ai go library lightrag llm
Total 62
30 days +0

Repos sorted by Jev.