About
llm-d Router is an intelligent entry point for LLM inference traffic, providing load and prefix-cache aware routing, request prioritization, and advanced flow control. It utilizes an Endpoint Picker (EPP) that integrates with proxies like Envoy via the ext-proc protocol. Supporting both Standalone and Kubernetes Gateway API modes, it enables performance-targeted scheduling, model name rewriting for canary rollouts, and coordinates complex multi-stage lifecycles like Prefill/Decode (P/D) disaggregation.
Features
- KV-Cache Aware Intelligent Routing
- Native Kubernetes Gateway API Integration
- Multi-level Request Prioritization
- Prefill/Decode Disaggregation Support
- Dynamic Model Rewriting & Canary Support
Supported Platforms
Links
Related AI Industry News
Creative Prompts Emerge as a Defense Against AI Job Applicants
As AI-driven automated job applications surge, recruiters are adopting unconventional tactics to filter candidates. By embedding creative prompts—such as asking for a whimsical poem about a frog—employers are finding effective ways to distinguish authentic human applicants from generic bot-generated responses.
Read Full Story →
llove v0.3.0a1: Building Customizable LLM Observability Dashboards in the Terminal
llove v0.3.0a1 introduces a high-performance terminal UI for LLM observability. Built on Textual, it eliminates the need for heavy browsers, offering customizable layouts via TOML and low-latency monitoring for SREs and high-security environments.
Read Full Story →
Snap, YouTube Settle School Social Media Suit Ahead of Trial
Snap and YouTube have reached a settlement in a major lawsuit led by school districts, which alleged the platforms' addictive nature harmed student mental health. The agreement was finalized just before the scheduled trial, highlighting the increasing legal pressure on social media companies regarding minor safety.
Read Full Story →