Local AI sovereignty
Keep the model on your side of the wire.
Every prompt you send to a hosted model crosses a boundary you don't own. Running the model yourself moves that boundary back to your hardware. Here's what that changes, what it costs, and what it doesn't fix.
Nothing on the inference path crosses the network.
Sovereignty is four separate controls
"Private AI" gets used loosely. It's worth separating what you actually hold onto, because a system can give you one of these and none of the others.
- Data
- What the model reads, and where it stops. Prompts, attached files, source code, customer records, log excerpts. On a local system all of it stays on the machine doing the work — there is no copy on someone else's disk to retain, subpoena, or leak.
- Access
- What the system is allowed to touch. Which directories, which hosts, which credentials, which network segments. You set the reach, and you can prove what it was by reading the config yourself.
- Behavior
- Which model, which version, which rules. A hosted model can change under you overnight and quietly break a workflow you tuned around it. Local weights don't move unless you move them.
- Continuity
- Whether it still works next quarter. No deprecation notice, no rate limit, no price change, no outage on someone else's status page. The system you set up is the system you keep.
Where your data goes, side by side
Neither column is wrong for every job. The point is to know which one you're choosing.
| Hosted API | Local model | |
|---|---|---|
| Prompt contents | Leave your network on every request | Never leave the machine |
| Files and context | Uploaded to a third party | Read from your own disk |
| Retention | Governed by a policy you can read but not enforce | Whatever your own logging keeps |
| Availability | Depends on their uptime and your link | Works with the network unplugged |
| Cost shape | Metered per token, forever | Hardware upfront, then electricity |
| Model changes | Can shift without warning | Only when you pull new weights |
| Rate limits | Enforced by the provider | Bounded by your own GPU |
| Peak capability | Frontier-class on the hardest tasks | Strong, and improving fast, but not frontier |
| Who maintains it | They do | You do |
What local AI doesn't fix
Anyone selling you local AI without this section is selling. Four things move onto your plate the moment you bring the model home.
The box is now the asset
You removed the exposure of data in transit and replaced it with a machine holding everything, sitting on your LAN. Disk encryption, account separation, patching, and physical access all matter more than they did before.
You're the one on call
Driver updates, VRAM exhaustion, a model that stops calling tools correctly after an upgrade — nobody else is going to notice or roll it back. Budget the maintenance honestly before you commit a workflow to it.
There is still a capability ceiling
Open-weight models have closed most of the everyday gap and keep closing it. On the hardest long-horizon reasoning they are not yet frontier. Match the model to the task instead of forcing one tool onto every job.
Local doesn't mean careful
A model with shell access on your own hardware can still delete the wrong directory. Sovereignty is about who holds the controls, not about whether the controls are needed — approval gates, scoped permissions, and backups still do the real work.
Who runs local, and why
Data that can't leave
Client records, case files, health or financial detail, internal source code. When the rule is "this doesn't go to a third party," a local model is the only version of the tool you're allowed to use.
Predictable cost
One machine, one power bill, no per-seat pricing and no surprise invoice when usage doubles. The tool you bought stays the tool you own.
Full access to the stack
Swap models, tune sampling, wire in your own tools, run the whole loop offline, and read every log. Nothing is behind an endpoint you can't inspect.
Works without a link
Remote sites, restricted networks, boats, vehicles, and anywhere the connection is slow, metered, or not permitted. Inference doesn't care that the uplink is down.
What it actually takes to run
The GPU is the constraint that matters. Everything else is about keeping it fed. These are general guidelines for local inference, not a hard floor.
| VRAM | Model class it fits | Realistic use |
|---|---|---|
| 8 GB | 7–8B at 4-bit quantization | Drafting, summarizing, quick code help, short context |
| 12–16 GB | 12–14B comfortably, or 8B with long context | Daily assistant work with room for tools and documents |
| 24 GB | 24–32B at 4-bit | The common sweet spot for agentic, tool-calling workloads |
| 48 GB+ | 70B class, or several models resident | Heavier reasoning, parallel jobs, vision plus text together |
- System RAM
- 32 GB is a comfortable floor; 64 GB gives room to run other work alongside the model. If layers spill out of VRAM, system memory absorbs them — slowly.
- Processor
- A modern multi-core CPU. It won't set your token rate when the model fits on the GPU, but it handles prompt processing, tooling, and everything the agent runs outside the model.
- Storage
- NVMe, with real headroom. A single quantized model file runs from a few gigabytes to forty, and you will end up keeping more than one. Fast storage also cuts load time between model swaps.
- Operating system
- Any modern 64-bit OS. Linux is the path of least resistance for drivers, containers, and unattended service management.
The terms you'll hit in the first hour
- Inference
- Running a trained model to get an answer. This is what your GPU is doing; training is a separate, far heavier job you're not signing up for.
- Weights
- The model file itself. Open-weight models are ones you can download and run without asking permission per request.
- Quantization
- Compressing weights to lower precision so they fit in less VRAM. Labels like Q4_K_M describe how aggressively. Lower precision means smaller and faster, with some quality traded away.
- VRAM
- Memory on the graphics card. The single biggest factor in which models you can run and how fast they respond.
- Offloading
- Pushing layers that don't fit in VRAM into system RAM. It keeps a big model running, and it is dramatically slower.
- Context window
- How much text the model can hold at once — your prompt, the conversation, and any files it read. Bigger windows cost memory.
- Token
- The chunk a model reads and writes in, roughly a short word or word-fragment. Speed is measured in tokens per second.
- Tool calling
- The model emitting a structured request to run something — read a file, run a command, query an API. It's the difference between a chatbot and an operator.
- RAG
- Retrieval-augmented generation: searching your own documents and handing the relevant passages to the model, so it answers from your material instead of memory.
- Egress
- Data leaving your network. The metric this whole page is about.
- Air gap
- A system with no network path outward at all. Local inference is what makes an air-gapped AI workflow possible.
- Agentic
- A model working in a loop — plan, act with tools, read the result, adjust — instead of answering once and stopping.
Frequently asked questions
What is AI sovereignty?
Why run AI locally instead of using a hosted API?
Is a local model as capable as a hosted frontier model?
What hardware do I need to run a local AI system like Horus?
Does local AI work without an internet connection?
Is local AI automatically more secure?
What does running local AI cost?
How does Horus fit into this?
Surface Vector Technologies
Horus: an AI operator that stays on your hardware
Most local AI setups stop at chat. Horus is built to do the work — read and edit files, run commands, drive a browser, and hold a job through to the end — while the whole loop stays inside your network.
- Runs on your machine, with no cloud egress on the inference path
- Agentic tool use across files, shell, browser, and scheduled jobs
- Human approval before it changes anything on disk
- Bring your own open-weight models and swap them whenever you want