K-AI

How K-AI works

Five flows cover the whole system: the architecture, a chat request, a rented pod, a document going into a knowledge base, and who is allowed to do what.

Architecture

Three kinds of caller reach K-AI: chat front ends and software over the OpenAI API, agents over MCP, and people in the control panel. Everything passes the edge, where people authenticate with single sign-on and software presents a scoped key. The router is the centre: it owns the queue, picks nodes and grounds chats in knowledge. Compute nodes run vLLM; the knowledge API stores documents in a graph and vector database.

ClientsEdgeK-AI control planeCompute (your nodes and rented pods)Knowledge storagechat APIknowledge APIground a chatChat front end(Open WebUI or any)Apps, scripts, SDKs(OpenAI-compatible)Agents over MCP(search your knowledge)Single sign-on for people · personal API keys with grants for softwareControl panelprofiles · nodes · keys · usageK-AI routerqueue · dispatcher · profiles · groundingKnowledge APIREST + MCP, per-base grantsHome GPU node(Linux, node agent)Docker node(Windows / Linux)RunPod podsstart on demand, stop idleKnowledge nodeembed + extract modelsGraph + vector databaseText extraction
Everything speaks the OpenAI API at the front and the router decides where each request runs.

A chat request

noyesyesnoyesnoyesnoChat request arrives(OpenAI-compatible API)Key valid?model allowed?401 / 403 refusedResolve the profile and itsknowledge bases (key grants apply)A node already servesthese launch settings?Send straight to that node(profiles sharing settings share it)An eligible node canload it (GPU fit)?Queue, pull if needed,swap the model inA rented pod could serve it?Start a pod: button, scheduleor burst; requests wait503 with the reasonno node can serve itRequestadmittedGround in knowledge:inject and/or search tool(fails open)Proxy to the node, streamthe answer, record usage
A request is authorised, routed to a node already serving the model when possible, and grounded in knowledge once it is admitted.
  1. Authorise. The router checks the bearer key (the shared key or a personal one), its expiry and that it may use models. Failures return 401 or 403 in the OpenAI error shape.
  2. Resolve the profile. The model name is a profile. K-AI works out its engine settings (the load key) and its knowledge bases, intersected with what this key may read. Bases the key can't read are skipped.
  3. Find a node. The dispatcher considers nodes that are online, allowed by the profile's placement rules, big enough for the model and not recently failed. It prefers one already serving the model, least busy first, local before remote.
  4. Queue and swap. If nothing is serving the model, the request joins a queue. When in-flight requests drain and the grace window passes, the node pulls the model if it lacks it, stops the old engine, starts the new one and waits until it is ready. One swap serves everyone waiting.
  5. Nothing can serve it? If a rented pod could, the 503 says so and points to Start a pod. Otherwise it names the reason for each node.
  6. Admit and ground. Once admitted, the router searches the knowledge bases (inject), offers a search tool (tool), or both, so a slow knowledge service can't hold up the queue. If the service is down the request proceeds without it.
  7. Serve and record. The request is proxied to the node, the stream passes through, and a usage row is written: tokens, latency, node and status, never the content.

A RunPod pod

yesnoyesStart a pod(button, or burst)Another podalready deploying?Say so and stop:no duplicate podsCreate the pod in thevolume's regionPull the engine image(~1.5 min)Load the model from thenetwork volume (~6 min)Pod joins the routeras a nodeServing requestsIdle for theconfigured time?Stop the pod(volume kept)Reload profile:change settings, no new podSpend cap reached?pods stop, volumes excluded
A cold start takes about 15 minutes; a warm reload about 2. Idle pods stop themselves.
  1. Start. A button press, a schedule window or a burst rule asks for a pod. K-AI refuses if the spend cap is reached, no pod image is configured, or a pod for that profile is already deploying.
  2. Check the cap. The new pod's hourly price plus what is already running, looked ahead one hour, must stay under the monthly cap.
  3. Create and boot. K-AI creates the pod in the network volume's region, trying GPU types in order. A boot timeout terminates pods that never come online.
  4. Join. The pod's agent comes up; K-AI sees it and adds it to the router as a node.
  5. Load. The model is read from the network volume (downloaded once if missing) and the engine starts. About 15 minutes cold, about two minutes on a warm reload.
  6. Serve. Requests arrive like any other node's, with local nodes preferred unless the profile says otherwise.
  7. Stop. After the idle period with no finished request the pod is deleted; the model stays on the volume. A spend-cap latch, a boot timeout, a 12-hour backstop or an operator delete also end a pod.

Knowledge ingestion

noyesSkip the duplicateUpload a file,URL or captureAlreadyindexed?Extract text(PDF, EPUB, office, HTML)Split into ~700-tokenchunksEmbed each chunk(router alias: embed)Extract entities andrelations (alias: extract)Write chunks, graph andvectors to the databaseJob done(progress is visible)Search: vector + keyword + graph → ranked passages
One job runs at a time per knowledge API; extraction on the knowledge node is the slow step.
  1. Accept. An upload, URL, chat capture or git sync arrives. Oversized files and duplicates (same hash in the same base) are rejected.
  2. Queue. A job is created; jobs run in order and resume after a restart.
  3. Extract text. Text and code are read directly; PDFs, EPUBs and Office files go through Apache Tika, with OCR for scanned pages. Headings are kept.
  4. Chunk. Heading-aware chunks of about 700 tokens with a small overlap.
  5. Embed. Each chunk gets a vector from the embed model through the router.
  6. Extract the graph. The extract model returns entities and relationships per chunk, each with a short reason. Invalid answers are retried once.
  7. Store. Chunks, vectors, entities and relations are written; evidence accumulates on repeated relations.
  8. Search. A query is embedded, matched by vector and by keyword, fused, and expanded one hop through the graph.
Speed: graph extraction is the slow step. On a single 16 GB consumer GPU, a typical technical book (a few hundred chunks) takes on the order of ten to fifteen minutes. Embedding is fast.

Sign-in and grants

yesyesnonoPerson opensthe control panelOIDC single sign-on(groups: users / admins)Mint a personal key:models, knowledge grants,expiryKey stored hashed;shown onceSoftware calls the APIwith the keyKey active andnot expired?Grant covers the modeland each knowledge base?Served; usagerecorded per keyRefuse (401/403);or drop the unreadable base
People sign in with SSO; software uses scoped keys that can only be narrowed, never widened.
  1. People sign in. The control panel is behind an OIDC reverse proxy. Membership of the allowed group lets you in; an admin group unlocks admin pages.
  2. Mint a key. On the API keys page, name it, choose model and knowledge grants and an expiry. The key is shown once; only a peppered hash is kept.
  3. Software calls the API. The router verifies the key and checks the grant. The knowledge API asks K-AI to verify the same key over a pinned TLS channel and caches the answer briefly.
  4. Narrow or revoke. Take effect at once on the router and within a minute on the knowledge API. Keys can only be narrowed.