Repositories

34

Total stars

18,464

Trending

vllm-project

An open, programmable decision layer for models and compute.

Created Aug 2025

ai-gateway guardrails inference kubernetes llm
Total 6,518
30 days +298
T-Crypt

Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.

Created Oct 2026

comfyui golang gpu inference llama-cpp
Total 300
30 days +275
theagentrouter

Manages Unified Access to Generative AI Services built on Envoy Gateway

Created Oct 2024

ai ai-gateway api-gateway envoy envoyproxy
Total 2,430
30 days +179
bryanwalter710-debug

Solana DeFi Security: On-Chain Governance Risk Analysis for Multisig, Timelock, and Upgrade Authorit

Created Nov 2026

angulardart ark btc cam curated-list
Total 153
30 days +153
ai-dynamo

Snapshot gets GPU pods ready in seconds instead of minutes — by restoring a fully initialized GPU worker instead of starting one from scratch. Kubernetes-native checkpoint & restore that runs alongside your existing stack.

Created Jul 2026

checkpoint-restore cold-start criu cuda-checkpoint gpu
Total 172
30 days +101
google

Agent Mesh

Created Apr 2026

a2a a2a-protocol agent-mesh agentic-ai inference
Total 1,207
30 days +85
llm-d

llm-d Router: The intelligent entry point for inference requests

Created May 2025

ai gateway-api inference kubernetes networking
Total 471
30 days +51
dsx-ai-factory

Platform for deploying and routing GPU-accelerated inference, streaming, and batch workloads at scale.

Created Apr 2026

cloud-functions gpu inference kubernetes nvidia
Total 270
30 days +31
defilantech

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.

Created Nov 2025

ai apple-silicon autoscaling edge-computing gguf
Total 267
30 days +30
utkuozdemir

Nvidia GPU exporter for prometheus using nvidia-smi binary OR using NVML

Created Jun 2021

ai cryptocurrency gaming inference llm
Total 1,598
30 days +29

Top

vllm-project

An open, programmable decision layer for models and compute.

Created Aug 2025

ai-gateway guardrails inference kubernetes llm
Total 6,518
theagentrouter

Manages Unified Access to Generative AI Services built on Envoy Gateway

Created Oct 2024

ai ai-gateway api-gateway envoy envoyproxy
Total 2,430
utkuozdemir

Nvidia GPU exporter for prometheus using nvidia-smi binary OR using NVML

Created Jun 2021

ai cryptocurrency gaming inference llm
Total 1,598
google

Agent Mesh

Created Apr 2026

a2a a2a-protocol agent-mesh agentic-ai inference
Total 1,207
onepanelio

The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.

Created Sep 2019

ai aiops annotation computer-vision deeplearning
Total 733
WaveSpeedAI

High-performance serverless GPU task orchestration — the scheduling layer behind wavespeed.ai

Created Oct 2025

go gpu inference orchestration scheduler
Total 635
kaito-project

🏗️ Fine-tune, build, and deploy open-source LLMs easily!

Created Sep 2023

ai buildkit chatgpt docker fine-tuning
Total 544
kubedl-io

Run your deep learning workloads on Kubernetes more easily and efficiently.

Created Dec 2019

container deep-learning inference kubernetes machine-learning
Total 534
llm-d

llm-d Router: The intelligent entry point for inference requests

Created May 2025

ai gateway-api inference kubernetes networking
Total 471
InftyAI

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

Created Nov 2023

huggingface inference inference-platform kubernetes llamacpp
Total 315
T-Crypt

Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.

Created Oct 2026

comfyui golang gpu inference llama-cpp
Total 300
ai-dynamo

Kubernetes enhancements for Network Topology Aware Gang Scheduling & Autoscaling

Created Sep 2024

agentic auto-scaling auto-scaling-group disaggregated gang-scheduling
Total 298
tensorchord

Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)

Created Jul 2023

cluster-manager hacktoberfest inference llm llmops
Total 282
dsx-ai-factory

Platform for deploying and routing GPU-accelerated inference, streaming, and batch workloads at scale.

Created Apr 2026

cloud-functions gpu inference kubernetes nvidia
Total 270
defilantech

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.

Created Nov 2025

ai apple-silicon autoscaling edge-computing gguf
Total 267
adammpkins

AI terminal assistant for any OpenAI-compatible API. Features interactive chat TUI, command generation, code explanation, and streaming responses. Works with Ollama, OpenAI, LM Studio, and more.

Created Aug 2023

ai inference llm python
Total 194
nikolaydubina

LLaMA-2 in native Go

Created Jul 2023

inference large-language-model llama llama2 llm
Total 191
ai-dynamo

Snapshot gets GPU pods ready in seconds instead of minutes — by restoring a fully initialized GPU worker instead of starting one from scratch. Kubernetes-native checkpoint & restore that runs alongside your existing stack.

Created Jul 2026

checkpoint-restore cold-start criu cuda-checkpoint gpu
Total 172
NexusGPU

Tensor Fusion is a state-of-the-art GPU virtualization and pooling solution designed to optimize GPU cluster utilization to its fullest potential.

Created Nov 2024

ai amd-gpu autoscaling dynamic-resource-allocation gpu
Total 171
pyadav

[deprecated] AI Gateway - core infrastructure stack for building production-ready AI Applications

Created Jan 2024

ai aistudio api gateway generative-ai
Total 162
bryanwalter710-debug

Solana DeFi Security: On-Chain Governance Risk Analysis for Multisig, Timelock, and Upgrade Authorit

Created Nov 2026

angulardart ark btc cam curated-list
Total 153
pmady

KEDA External gRPC Scaler for GPU workloads - native NVML metrics via DaemonSet, no Prometheus required

Created Mar 2026

ai-infrastructure autoscaling daemonset gpu gpu-autoscaling
Total 136
ai-runway

✈️ Kubernetes-native platform for deploying and managing AI inference across multiple providers

Created Dec 2025

ai cloudnative dynamo inference kubernetes
Total 103
dsx-ai-factory

Translation layer that maps any Kubernetes framework's Custom Resource Definitions (CRDs) into a standardized, generic structure.

Created Jul 2025

crd inference k8s karta kubernetes
Total 100
llmariner

Extensible generative AI platform on Kubernetes with OpenAI-compatible APIs.

Created Mar 2024

ai autoscaling fine-tuning gpu inference
Total 97
ShadowSafin

AI inference gateway and control plane for multi-provider routing, policies, caching, tools, and observability.

Created Oct 2026

ai ai-gateway api-gateway inference llm
Total 93
greynewell

Route inference across providers.

Created Sep 2024

ai-gateway ai-infrastructure anthropic api-gateway cost-tracking
Total 93
sjkaliski

🔮 Use TensorFlow models in Go to evaluate Images (and more soon!)

Created Mar 2018

go golang inference machine-learning prediction
Total 63
jitllm

Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory

Created Sep 2026

cuda inference inference-acceleration inference-engine inference-server
Total 58
canonical

Local inference, optimized for your hardware

Created Jun 2025

ai inference llm local-ai model-serving
Total 55
songtianyi

go binding for mxnet c_predict_api to do inference with pre-trained model

Created Dec 2016

cgo deep-learning golang inference machine-learning
Total 54
OpenCSGs

CSGLite is a lightweight local LLM inference platform. One command downloads, loads, and chats with models. It ships a web UI, OpenAI-compatible API, llama.cpp inference, resumable downloads, marketplace browsing, and one-click AI app and coding-agent setup—all in a single cross-platform binary.

Created Mar 2026

ai-gateway cc-switch inference ollama-api
Total 53
scitix

Arks is a cloud-native inference framework running on Kubernetes

Created Apr 2025

ai cloudnative-services dynamo inference kubernetes
Total 51

Related

All topics →