llm-d

llm-d

GitHub

Repositories

4

Total stars

972

Top repositories

llm-d Router: The intelligent entry point for inference requests

Created May 2025

ai gateway-api inference kubernetes networking
Total 471

A lightweight, configurable, and real-time simulator designed to mimic the behavior of vLLM without the need for GPUs or running actual heavy models.

Created Apr 2025

incubating
Total 243

Distributed KV cache scheduling & offloading libraries

Created Apr 2025

ai incubating
Total 197

Variant optimization autoscaler for distributed inference workloads

Created Jun 2025

Total 61