Self hosted AI for business: hardware, upkeep and which jobs fit

Self hosted ai for business means the model runs on a computer you own or control, in your building or a rack you rent, instead of on a provider's servers. For a small service company it is a real option, but not a default one. Most first builds start in the cloud. This page covers what the hardware has to do, what it costs you in upkeep, and which jobs are worth keeping on-site.

What self-hosting actually means

There are three setups, and people mix them up.

• Cloud: every request goes to a provider's model over the internet.
• Self-hosted (on-premise): the model files sit on your own machine and answer requests there.
• Hybrid: routine work runs locally and a hard job calls a cloud model when you allow it.

For a software tool such as Ollama, the first claim is simple: Ollama runs locally, and its makers say they do not see your prompts or data when you run it that way. That is the privacy argument in one line. Nothing about it fixes a badly connected system, but it does mean the text of a customer record does not leave your network for that job.

What the hardware has to do

The main constraint is memory. A model has to fit in the graphics card's memory (VRAM) to run at a good speed. Ollama's own library lists Llama 3.1 at 4.9GB for the 8 billion parameter version and 43GB for the 70 billion version, both with a 128K context window. Those are file sizes before the system needs room to work.

What follows from that:

• A small model on one decent graphics card can handle routine jobs: summarizing a day of jobs, drafting a short text, classifying an inbound message.
• A large model needs far more memory than a typical office PC has. A 43GB model will not fit on a single consumer card. Ollama's documentation notes that when a model does not fit, it loads partly onto the GPU and partly into system memory, which is slower, and that it can spread a model across several GPUs.
• Requests queue. Ollama processes queued requests in order. If three agents ask at once, two wait. That is fine for an overnight job and wrong for a phone line.

We are not quoting a hardware price or a speed figure here. They depend on the model, the card and the job, and they change every few months. Anyone who gives you a number before asking what the machine will do is guessing.

What it costs you in upkeep

Self-hosting moves work onto you.

1. Someone updates it. Models, the runtime and the operating system all change.
2. Someone secures it. Ollama binds to 127.0.0.1 by default and you have to deliberately change a setting to expose it to your network. If an agent on another machine needs it, the network path is something a person must lock down.
3. Someone watches it. A server that fails at 2am and quietly skips the morning report is worse than a cloud service with an outage page.
4. Someone owns the hardware. Power, cooling, a place to put it, a plan for when the card dies.

Which jobs are worth keeping on-site

Good fits: work that runs every night or every morning whether anyone is in the building, because per-use cloud pricing is a poor match for a fixed daily job. Routine checks on records the owner would rather not send out, such as customer names, invoices and repair order notes. Work that should not stop because a provider is having a bad day.

Poor fits: an occasional hard task, like a messy customer email that needs a stronger model. Send that to a capable cloud model and pay for what you use. A single HVAC or plumbing company with two follow-up workflows rarely needs a server at all.

Where small businesses are today

You do not need to settle this to get started. The Census Bureau puts business AI use at 19.8% as of May 3, 2026. The Federal Reserve's survey of small employers found that time to implement and train employees was named by 37% of firms planning to start. A hardware project adds to that time. Starting in the cloud and moving a job on-site later is a normal path.

How Cortex handles it

Cortex designs the setup around the workload: cloud, private hardware or a mix. Our private AI page explains the split. Routine overnight checks and briefings can run on local models, rules handle what does not need AI, and a cloud model is called when a task requires it. At Detail Driven, we run local models for the overnight checks and morning briefing on dedicated Linux hardware the owner controls, and call a cloud model only when a job needs more. For a longer comparison, see local vs cloud AI for service businesses and the automotive private AI workforce write-up.

What we do not do: tell you to buy a server so the proposal looks bigger. If your workload does not call for hardware, we say so.

Next step

Book the free AI Operations Audit. It tells you which of your jobs belong in the cloud, which could run on-site and which need no AI at all. You owe us nothing after it.