Everything in the box
K-AI is a control panel and a router for self-hosted inference, with a knowledge base built in. Here is what each part does.
Routing and queueing
OpenAI-compatible
Chat completions, completions and embeddings on /v1, plus a model list. Point any OpenAI client at K-AI and use a profile name as the model.
Requests wait, they don't fail
If the model isn't loaded yet, the request joins a queue. Streaming clients get keepalive comments showing the queue position and whether the model is loading or being pulled.
Fair swaps
A model swap starts after in-flight requests drain plus a short grace window, and one swap serves every request waiting for that model. A busy model can't starve the others.
Shared loads
Profiles whose engine settings are identical share one loaded model. Switching between them, for example the same model with different knowledge bases, needs no restart.
Honest errors
When nothing can serve a model you get a 503 that names the reason per node: not enough VRAM, offline, excluded after a failed load.
Usage-safe streaming
Streams are recorded with token counts without changing what your client sees.
Profiles and the model finder
A profile is a small settings file kept in a git repository, so every change is a commit with an author. It captures the model, GPU count, context length, quantisation, cache type, tool-call and reasoning parsers, where it may run, how it may use rented GPUs and which knowledge bases ground it.
Model finder
Search Hugging Face from the panel, filter by format (4-bit, FP8), and see a verdict for your hardware: fits, tight, too big. Download with one click; gated models work with a token.
Fit estimates
K-AI estimates weights plus KV cache against your GPUs and recommends a context length. Models with no local files are estimated from the Hugging Face hub.
Tool calling
All of vLLM's tool-call parsers can be selected, grouped by model family with sensible suggestions, plus reasoning parsers for thinking models.
Nodes and fleet routing
Node agent
A small agent on each host reports GPUs, memory, disk and cached models, pulls models, and starts and stops the engine in a container.
Linux and Windows
Add a Linux host from the panel (K-AI checks the OS and GPU, installs Docker and the NVIDIA toolkit, and deploys the agent). Windows hosts run Docker Desktop with a one-shot bootstrap script.
Smart dispatcher
A request goes to a node already serving the model, preferring the least busy and local over remote. If that node backs up, a second replica loads on an idle node.
Knowledge nodes
A node can take the knowledge role and run an embedding model and an extraction model side by side on one GPU, so a single gaming PC can power your knowledge bases.
Node groups
Combine several nodes into a Ray group for multi-GPU models through pipeline parallelism. auto mode gated
Failure handling
A failed load restores the previous model, excludes that node for ten minutes and, if possible, retries elsewhere. Waiting requests move when a node goes offline.
RunPod burst, idle stop and the spend cap
Rent GPUs only while they earn their keep. K-AI creates single-GPU pods through RunPod, treats each as a node, and removes it when it goes quiet.
Start by button, schedule or burst
Start a pod from a profile or the picker, which shows live GPU prices, stock and which profiles fit. Burst rules start one when requests are waiting; schedule windows start one on a timetable.
Network volumes
Models live on a RunPod network volume per region, so a fresh pod loads weights in minutes instead of downloading them. Volumes are excluded from the spend cap if you choose.
Idle stop
A pod is deleted after a configurable idle period with no finished requests; models stay on the volume. Backstops delete pods stuck booting, running too long, or that K-AI doesn't know about.
Spend cap
A monthly cap with an 80% warning and a 100% latch that stops every pod and refuses new ones. Each start is checked against the cap first.
No duplicate pods
If a pod for a profile is already deploying, K-AI says so instead of starting another.
Reload profile
Change a profile and press Reload: the running pod reloads with the new settings in about two minutes, with no new pod and no volume re-download.
Knowledge bases
The knowledge base turns documents and git repositories into searchable text, vectors and a graph of entities and relationships.
Many formats
PDF, EPUB, Office files, email, HTML, text and code. Scanned PDFs fall back to OCR. Spreadsheets become readable per-sheet sections. Files up to a configurable size; duplicates are skipped by hash.
Graph, not just chunks
Every chunk is read by an extraction model that records entities and relationships with a one-sentence reason. Search expands from matching passages to what they connect to.
Hybrid search
Vector search and keyword search are fused, then expanded one hop through the graph and returned as ranked passages, entities, relations and a ready-to-use context block.
Code and git
Track a repository over https or ssh with polling and signed webhooks. Code is parsed by tree-sitter into symbols with calls, imports and inheritance edges and cached summaries.
REST, OpenAPI and MCP
Use the knowledge base from scripts or from agents over MCP: list bases, search, explain a relationship, find a symbol, walk a symbol's neighbours.
Panel workflow
Drag-and-drop upload, document status and re-extract, move between bases, a search box to test retrieval, and an inbox for captured chat attachments awaiting review.
Grounding modes
Bind up to eight knowledge bases to a profile and pick how the model uses them.
| Mode | What happens |
|---|---|
inject | Before every answer K-AI searches and appends the passages to the last user message, which keeps the system prompt and history cache-stable. |
tool | A search_knowledge function is added; the router runs searches the model asks for, up to three rounds. Needs a tool parser. |
both | Inject, and also offer the tool. Best for small models, which often skip the tool. |
off | No grounding. |
Grounding fails open: if the knowledge service is slow or down, the answer still arrives, with a header saying knowledge was unavailable. A request can override the bases and mode in its body, and the response reports what happened.
Nightly upgrades
Daytime ingest uses a small, fast extraction model. In a quiet window K-AI can switch idle knowledge nodes to a larger extraction model and re-extract the backlog, replacing a chunk's graph only when the new result succeeds. Windows, idle period and history are configurable, and admins can run a night on demand.
API keys and grants
- Sign in with your own single sign-on; create personal keys on the API keys page.
- Each key carries grants: whether it may use models, and a knowledge level (read, read-write, admin) over named knowledge bases or all of them.
- Keys have an expiry (30, 90, 365 days or none), can be narrowed after creation but never widened, and are shown once. Only a peppered hash is stored.
- The same key works for the model API, the knowledge REST API and MCP. A knowledge base a key can't read looks identical to one that doesn't exist.
- Revoking takes effect immediately on the router and within a minute on the knowledge API.
Usage
One row per request: when, who or which key, client, API, model, profile, node, status, latency and token counts. K-AI never stores prompts, completions, document text or search queries. People see their own usage, admins see everyone's, and you can export CSV. Rows are kept for 90 days.
Security
Sign-in
The panel sits behind an OIDC reverse proxy with group checks, short sliding sessions and idle sign-out. Management writes are accepted only from that proxy.
Encrypted credentials
Git tokens, registry and cloud API keys live in an AES-256-GCM store bound to their name, host and consumer, and are never shown back or written to logs.
Hardened by default
Constant-time token checks, SSRF guards on URL and git ingest, signed webhooks, pinned container images and a pod-side gate that checks keys even where the engine doesn't.