GitHub topics

vllm

Mostly in LLM Infrastructure 56%

Repositories

23

Total stars

32,582

Trending

vllm-project

An open, programmable decision layer for models and compute.

Created Aug 2025

ai-gateway guardrails inference kubernetes llm
Total 6,518
30 days +298
T-Crypt

Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.

Created Oct 2026

comfyui golang gpu inference llama-cpp
Total 300
30 days +275
kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

Created Mar 2019

artificial-intelligence cncf genai hacktoberfest istio
Total 6,258
30 days +125
loxilb-io

Inference-aware L4/L7 load balancer for LLM serving fleets, forked from loxilb

Created Jul 2026

ai-gateway ebpf kubernetes kv-cache llm-inference
Total 71
30 days +66
defilantech

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.

Created Nov 2025

ai apple-silicon autoscaling edge-computing gguf
Total 267
30 days +30
halfrost

✍🏻 Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field 冰霜之地:源码解析、系统设计与工程实践笔记

Created Apr 2017

algorithms blog cryptography go golang
Total 13,269
30 days +23
thushan

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.

Created May 2025

ai amd golang intel llama-cpp
Total 338
30 days +20
matrixhub-ai

An Open-source, self-hosted AI model hub with Hugging Face compatibility, accelerating vLLM/SGLang performance.

Created Jan 2026

artificial-intelligence dynamo huggingface kubernetes llm
Total 366
30 days +13
ome-projects

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

Created May 2025

deepseek k8s kimi-k2 llama llm
Total 529
30 days +11
kubeai-project

AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.

Created Oct 2023

ai autoscaler faster-whisper inference-operator k8s
Total 1,294
30 days +11

Top

halfrost

✍🏻 Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field 冰霜之地:源码解析、系统设计与工程实践笔记

Created Apr 2017

algorithms blog cryptography go golang
Total 13,269
vllm-project

An open, programmable decision layer for models and compute.

Created Aug 2025

ai-gateway guardrails inference kubernetes llm
Total 6,518
kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

Created Mar 2019

artificial-intelligence cncf genai hacktoberfest istio
Total 6,258
kubeai-project

AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.

Created Oct 2023

ai autoscaler faster-whisper inference-operator k8s
Total 1,294
bricks-cloud

🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, vLLM, and open-source LLMs.

Created Jul 2023

ai anthropic api artificial-intelligence azure
Total 1,234
raids-lab

Crater is a cloud-native AI training & inference platform.

Created Jun 2025

buildkit deep-learning envd fumadocs helm-charts
Total 579
ome-projects

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

Created May 2025

deepseek k8s kimi-k2 llama llm
Total 529
matrixhub-ai

An Open-source, self-hosted AI model hub with Hugging Face compatibility, accelerating vLLM/SGLang performance.

Created Jan 2026

artificial-intelligence dynamo huggingface kubernetes llm
Total 366
thushan

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.

Created May 2025

ai amd golang intel llama-cpp
Total 338
InftyAI

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

Created Nov 2023

huggingface inference inference-platform kubernetes llamacpp
Total 315
T-Crypt

Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.

Created Oct 2026

comfyui golang gpu inference llama-cpp
Total 300
defilantech

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.

Created Nov 2025

ai apple-silicon autoscaling edge-computing gguf
Total 267
praetorian-inc

Simple LLM service identification - translate IP:Port to Ollama, vLLM, LiteLLM, or 60+ other AI services in seconds

Created Jan 2026

ai-security attack-surface capability golang llm-evaluation
Total 255
lordmathis

Unified management and routing for llama.cpp, MLX and vLLM models with web dashboard.

Created Jul 2025

llama-cpp llama-server llamacpp llm llm-inference
Total 162
voidmind-io

Privacy-first LLM proxy and AI gateway - load balancing, multi-provider routing, API key management, usage tracking, rate limiting. Self-hosted. Zero knowledge of your prompts.

Created Mar 2026

ai ai-gateway anthropic api-gateway docker
Total 149
pmady

KEDA External gRPC Scaler for GPU workloads - native NVML metrics via DaemonSet, no Prometheus required

Created Mar 2026

ai-infrastructure autoscaling daemonset gpu gpu-autoscaling
Total 136
openziti

Zero trust LLM gateway. OpenAI-compatible proxy with semantic routing and load balancing across OpenAI, Anthropic, Ollama, vLLM, and any compatible backend. Identity-based access, virtual API keys, and end-to-end encryption via OpenZiti

Created Feb 2026

agentic-ai ai-gateway ai-infrastructure ai-security anthropic
Total 109
ai-runway

✈️ Kubernetes-native platform for deploying and managing AI inference across multiple providers

Created Dec 2025

ai cloudnative dynamo inference kubernetes
Total 103
llmariner

Extensible generative AI platform on Kubernetes with OpenAI-compatible APIs.

Created Mar 2024

ai autoscaling fine-tuning gpu inference
Total 97
loxilb-io

Inference-aware L4/L7 load balancer for LLM serving fleets, forked from loxilb

Created Jul 2026

ai-gateway ebpf kubernetes kv-cache llm-inference
Total 71
scitix

Arks is a cloud-native inference framework running on Kubernetes

Created Apr 2025

ai cloudnative-services dynamo inference kubernetes
Total 51

Related

All topics →