LLM Infrastructure
Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.
Repos
200
Stars
636,933
Last 30 days
+26,248
An open-source engine that turns continuous content streams into search and monitoring. Durable ingestion, hybrid search, alerts, and plugins for formats, models and business rules.
Created May 2023
An open, programmable decision layer for models and compute.
Created Aug 2025
Reliable model swapping for any local AI server. Chosen by thousands of users. Batteries included. If we don't have it, you probably don't need it.
Created Oct 2024
Manages Unified Access to Generative AI Services built on Envoy Gateway
Created Oct 2024
Ultrafast serverless GPU inference, sandboxes, and background jobs
Created Nov 2023
🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, vLLM, and open-source LLMs.
Created Jul 2023
✨Reverse-engineered API for Gemini web app. It can be used as a genuine API key from OpenAI, Gemini, and Claude.
Created Dec 2025
🏗️ Fine-tune, build, and deploy open-source LLMs easily!
Created Sep 2023
Cut LLM costs by up to 80% and unlock sub-millisecond responses with intelligent semantic caching.A drop-in, provider-agnostic LLM proxy written in Go with sub-millisecond response
Created Nov 2025
An Open-source, self-hosted AI model hub with Hugging Face compatibility, accelerating vLLM/SGLang performance.
Created Jan 2026
A workload for deploying LLM inference services on Kubernetes
Created Aug 2025
High-performance Go proxy gateway for 9Router LLM routing — 32K RPS, 42MB RAM, CGO-free
Created Jul 2026
llama.cpp/ik_llama.cpp launcher: loads big MoE models across mismatched multi-GPU rigs by exact VRAM math.
Created Mar 2026
Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)
Created Jul 2023
A lightweight, production-ready RAG (Retrieval Augmented Generation) library in Go.
Created Jul 2024
JoyCode API 代理 — 协议翻译成 Anthropic/OpenAI 兼容格式,让 Claude Code 直接调用 JoyCode支持的模型,比如GLM 5.1等
Created Apr 2026
An open-source, cloud-native, high-performance gateway unifying multiple LLM providers, from local solutions like Ollama to major cloud providers such as OpenAI, Groq, Cohere, Anthropic, Cloudflare and DeepSeek.
Created Jan 2025
Unified management and routing for llama.cpp, MLX and vLLM models with web dashboard.
Created Jul 2025
Browserless Chrome uTLS + HTTP/2 OpenAI-compatible API daemon for Google Gemini Web (/v1/chat/completions & /v1/responses)
Created Oct 2026
Privacy-first LLM proxy and AI gateway - load balancing, multi-provider routing, API key management, usage tracking, rate limiting. Self-hosted. Zero knowledge of your prompts.
Created Mar 2026
将 Qoder 的 AI 服务转化为兼容 OpenAI/Claude/Codex 格式的 API,支持每日签到领 Credits、多账号管理、模型上下文展示
Created Jul 2026
Selfhost modern LLM stacks. Run the whole fleet from your terminal
Created Jun 2026
AI inference gateway and control plane for multi-provider routing, policies, caching, tools, and observability.
Created Oct 2026
Route inference across providers.
Created Sep 2024
Inference Hub for AI at Scale
Created Sep 2023
🚀 Free Unlimited OpenAI-compatible API for Google Gemini 4 Hydrogen & Neon. Supports Deep Thinking controls (High/Medium/Low), Agentic Task Mode, HTTP/3 QUIC, and multi-account pooling.
Created Feb 2026
Self-hosted voice gateway in Go that speaks the OpenAI Realtime (GA) WebSocket protocol and runs a cascaded ASR → LLM → TTS pipeline behind it. Keep clients unchanged while swapping speech and LLM providers at runtime
Created Sep 2026
A client/server for LLaMA (Large Language Model Meta AI) that can run ANYWHERE.
Created Mar 2023
Convert Google Gemini web into OpenAI-compatible API. Zero cost, single static binary.
Created Aug 2026
Multi-Provider AI Gateway, "Because we have LiteLLM at home" No personal logs by design. Model autodiscovery, Failover groups, High availability, Android companion app, and more!
Created Apr 2026
OpenAI-compatible multi-channel LLM relay gateway with admin console
Created Jul 2026
Local inference, optimized for your hardware
Created Jun 2025
Stop paying for AI APIs during development. LocalCloud runs everything locally - GPT-level models, databases, all free.
Created Jun 2025
A Go library implementation of LightRAG - an advanced Retrieval-Augmented Generation (RAG) system that uniquely combines vector databases with graph database relationships to enhance knowledge retrieval.
Created Mar 2025 Archived
Repos sorted by Jev.