K-AI documentation
What the pieces are, how they behave, and where the edges are. Installation guides are coming soon.
Overview
K-AI is a self-hosted inference provider: a control panel plus an OpenAI-compatible router in front of a fleet of vLLM GPU nodes, and a knowledge base (a graph and vector store) that any model can be grounded on. Clients call one /v1 endpoint with a model name that is a profile; K-AI decides which node serves it, queues the request, loads the model if needed, can rent a GPU on RunPod under a monthly cap, and can inject or expose knowledge search. People get scoped, revocable API keys; usage is recorded per request without storing content.
Naming
K-AI was called llm-ctl. The project name is K-AI; the software components, container images, environment variables and design specs still use their working names:
llm-ctl (control panel and router), kb-api (knowledge service) and node-agent. Where this page says "the router" or "the knowledge API" it means those.Components
| Component | What it is |
|---|---|
Control panel and router (llm-ctl) | One Python (FastAPI) process: a server-rendered panel, the OpenAI-compatible router, the scheduler and dispatcher, the node registry, the RunPod controller, the encrypted credential store, personal API keys and the usage store. The panel has tabs for status, model search, profiles, knowledge, jobs, nodes (with RunPod and groups), credentials, API keys, usage and connection help. |
Knowledge service (kb-api) | FastAPI with SurrealDB (HNSW vectors and full-text), Apache Tika, tree-sitter and an MCP server. Runs ingest jobs, hybrid search, graph queries, git source sync and the nightly backlog. It hosts no models: embedding and extraction go through the router's embed and extract aliases. |
Node agent (node-agent) | A small service on each host (bearer-token protected) that reports hardware and cached models, pulls models, and starts and stops engine containers. Variants: Linux (deployed over SSH from the panel), Windows (Docker Desktop bootstrap script) and the RunPod pod image. |
| Engines | vLLM, pinned to a tested version, started by the agent. Chat models, embedding models and extraction models are all vLLM. |
| Chat front end | Not part of K-AI. Open WebUI works well: point it at the router with a personal key and each chat profile appears as a model. |
| Single sign-on proxy | An OIDC reverse proxy (oauth2-proxy) in front of the panel. Any OIDC provider with a groups claim works. |
Concepts
| Term | Meaning |
|---|---|
| Profile | A named set of engine and policy settings. The profile name is the model id clients use. |
| Load key | A profile's engine arguments minus the served name. Profiles with equal load keys share one loaded model; switching between them never restarts the engine. |
| Node | A machine or pod that can run an engine. Kinds: Docker host, group member, knowledge, RunPod. |
| Knowledge node | A node that runs the embed slot and extract slot for knowledge bases instead of chat. |
| Knowledge base | A named collection of documents and git sources, searched together. |
| Grounding | Using knowledge-base search to inform a chat answer, by injection, a tool or both. |
| Grant | What a key may do: use models; and read, read-write or admin on named knowledge bases. |
| Burst | Starting a pod automatically because requests are waiting. |
Profile fields
Profiles are KEY=value files in a git repository; fields are written only when they differ from the default.
| Field | Purpose |
|---|---|
MODEL_PATH, SERVED_NAME | The model and the name the engine serves it under. |
PP_SIZE | Number of GPUs (pipeline parallel). |
MAX_MODEL_LEN, GPU_MEM_UTIL, KV_CACHE_DTYPE, QUANTIZATION | Context length, memory share (default 0.90), cache type (auto or fp8) and quantisation. |
EXTRA_ARGS, CUDA_GRAPH_SIZES | Extra engine arguments (for example a reasoning parser); fewer graph sizes start faster. |
TOOL_PARSER, TOOL_CHAT_TEMPLATE | Tool-calling parser and template. |
ROLE | chat, extract or embed. The last two run only on knowledge-node slots. |
KNOWLEDGE, KNOWLEDGE_MODE, KNOWLEDGE_K | Up to eight bases, grounding mode (off, inject, tool, both) and passages per lookup (1–20, default 6). |
LOCAL_ONLY, MIN_VRAM_GB, GPU_MATCH, NODES | Placement rules. |
REMOTE, RUNPOD_GPUS, RUNPOD_CLOUD, RUNPOD_REGION | Whether and where rented pods may serve it (never, allowed, preferred). |
IDLE_STOP_MIN, BURST_WAITING, BURST_AGE_S, BURST_MAX, SCHEDULE | Idle timeout, burst rules and schedule windows. |
API
Router (OpenAI-compatible)
| Endpoint | Notes |
|---|---|
POST /v1/chat/completions | Streaming and non-streaming. model is a profile. Optional body field knowledge: {"kbs": [...], "mode": "...", "k": 6} overrides grounding for this request and is stripped before forwarding. |
POST /v1/completions, POST /v1/embeddings | Proxied the same way. |
GET /v1/models | Every valid profile, with whether and where it is loaded. |
Authenticate with Authorization: Bearer <key>. Response headers: X-Node (which node served it) and X-Knowledge (what grounding did). Errors use the OpenAI error shape: 401 for a bad key, 403 for a missing grant, 404 for an unknown model, 503 when no node can serve, 504 on queue timeout.
curl https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer $K_AI_KEY" -H "Content-Type: application/json" \
-d '{"model":"my-profile","messages":[{"role":"user","content":"Hello"}]}'Knowledge API
| Endpoint | Purpose |
|---|---|
GET/POST /v1/kbs | List and create knowledge bases. |
POST /v1/kbs/{kb}/documents | Multipart upload (field files), up to 20 files per request. |
POST /v1/kbs/{kb}/ingest | Ingest text or a URL. |
GET /v1/jobs, /v1/jobs/{id} | Ingest and maintenance jobs with progress. |
POST /v1/search | Hybrid search in a base; returns passages, entities, relations and a context block. |
/mcp | MCP (streamable HTTP): list_knowledge_bases, search_knowledge, explain_relationship, find_symbol, code_neighbors. |
/openapi.json | The full REST description. |
Knowledge base details
- Chunking: heading-aware, about 700 tokens with 80 overlap, measured with the embedding model's tokenizer. Code is one chunk per symbol.
- Embeddings: 1024 dimensions, batches of 32, through the router alias
embed. - Graph: an entity is keyed by type and normalised name (exact merge only). Relations carry a kind, a short reason, evidence chunks and a weight. Deleting a document prunes its evidence and any orphaned entities and edges.
- Search: top candidates from vector and keyword search fused with reciprocal rank fusion, then a one-hop graph expansion, returning a context block of roughly 3,000 tokens.
- Git sources: https or ssh, credentials from the encrypted store, polling (default 15 minutes) and signed webhooks (HMAC for Forgejo, Gitea and GitHub). Limits: 2 GB and 50,000 files per repository, files over 1 MB and binaries skipped.
- Chat capture: Open WebUI's external content-extraction engine can point at the knowledge API, which extracts the file and queues it into a reserved
inboxbase for review. The inbox can't be bound to profiles or covered by wildcard grants.
RunPod
- Pod image: a K-AI agent plus vLLM, pushed to a registry RunPod can pull from. Deploy is disabled until one is configured.
- Volumes: one network volume per region, mounted at
/models, holds the model cache. K-AI never creates volumes. - Networking: through RunPod's HTTPS proxy (the default), or a WireGuard hub if you run one. Inside the pod the engine listens on loopback and the agent serves it behind a key check.
- Secrets: the pod receives an agent token and an engine key; a Hugging Face token is passed only if you enable it.
- Backstops: boot timeout, a pod-side self-delete timer, a stray-pod sweep and re-adoption of pods after a restart.
Defaults and limits
| Setting | Default |
|---|---|
| Swap grace window | 30 s |
| Queue timeout | 900 s (504 afterwards) |
| Streaming keepalive | every 10 s |
| Knowledge search timeout (inject) | 5 s, then fails open |
| Tool rounds | 3 |
| Node poll / offline after | 15 s / 3 misses |
| Spill-over to a second replica | more than 4 waiting or oldest waiter over 30 s; at most 2 replicas |
| Load-failure exclusion | 600 s |
| RunPod monthly cap | $50 (editable); warn at 80%, latch at 100% |
| RunPod idle stop | 15 min (per profile) |
| Pod boot timeout / max pod life | 15 min / 12 h |
| Personal keys | up to 20 active per person; expiry 30, 90, 365 days or none |
| Usage retention | 90 days |
| Panel session | 15 min sliding, signs out after 15 min idle |
| Knowledge upload | configurable per-file cap, 20 files per request |
Known limitations
K-AI is young and honest about its edges.
- Pods are not request-triggered. A chat request alone returns a 503; a button, schedule or burst rule starts pods. Cold start is about 15 minutes.
- Single-GPU pods only. No multi-GPU RunPod pods, knowledge slots on pods or Serverless.
- Not built yet: per-user quotas and rate limits, request priorities, a queue that survives restarts,
/v1/responsesproxying, hot-swap sleep mode for single-GPU nodes, and a user interface for per-base nightly settings. - Built but off by default: automatic Ray group mode and the fleet-wide embedding-model switch.
- Small models and knowledge: small models often skip the search tool and use injected context poorly. Prefer
bothmode and models of about 4B parameters or more. - Graph quality: no fuzzy entity merging and no re-ranker; a small extractor produces some trivial relations. OCR is English-only and for PDFs.
- Access control is per key and per knowledge base, not per document.
- Profile changes don't reach a pod that is already serving until you press Reload profile.
- Backups: the knowledge database has no built-in backup job yet; back up its storage yourself.
- Fit estimates are estimates, not guarantees.