The Fifty Example Test: how to test a local LLM on your own jobs

This is the guide behind the video. It is the grading sheet, the jobs to start with and the decision rule, so you can run the test this week without a developer deciding for you.

The test in two steps

1. Collect fifty real examples of one job. Fifty job notes to summarize, fifty voicemail transcripts to pull fields from, or fifty inbound messages to sort. Real ones from your own system, not made-up samples.
2. Run them through a local model and a cloud model, and grade every answer. The person who owns that work in your office marks each one fine, fixable or wrong.

Then one rule on top of both: a reply can fit the format perfectly and still hold the wrong address, so a person stays on anything a customer sees.

The reason to test instead of reading a ranking is simple. Model rankings change every few months, and none of them were run on your job notes. Anthropic's own guidance on building evaluations says to design tests that mirror your real-world task distribution and to include edge cases, such as ambiguous inputs where even people would disagree. Fifty examples from your own files do exactly that.

Who this is for

An owner or office manager at a service business who has heard "run AI on your own hardware" and wants to know which jobs that actually covers. You do not need a server yet. You need one job that repeats, fifty examples of it, and an hour of the right person's time to grade. If you want the background first, our guide to a local AI system for business explains what local models are good at.

What you need before you start

• One job, written in one sentence. "Summarize last night's job notes into three lines for the morning meeting." Not "use AI for the office."
• A local model running. Ollama is a common way to run one. Its FAQ says it does not see your prompts or data when you run it locally and binds to 127.0.0.1 by default, so nothing outside that machine can reach it until you change a setting.
• A cloud model you are allowed to use for this data. Check your provider's business terms before you paste customer records into anything.
• The grader. The person who does this job today. Not the person most excited about AI.
• A spreadsheet with the columns below.

Step 1: collect the fifty examples

Pick the job from the three kinds that tend to fit a local model, because the input is text you already have and the output is short or structured:

Job type	Example from a service business	What a good answer looks like
Summaries	Last night's job notes, a long email thread	Three lines, nothing invented, names and addresses right
Extraction into fields	Customer, address, system type and urgency from a voicemail transcript	Every field filled correctly or left blank
Routing and sorting	Spam, new lead, existing customer, scheduling question, complaint	The right label, or "unsure"

Ollama's Llama 3.1 page lists long-form text summarization among its use cases, with a 128K context window on every size, so summaries are a fair first test.

Pull fifty in a row from a normal stretch of weeks, not the fifty easiest. Include the messy ones. Remove card numbers and anything you would not want on a laptop.

Step 2: run both models and grade every answer

Give both models the same instructions and the same output format. For extraction jobs, ask for a fixed set of fields. Ollama can enforce a JSON schema on the reply, and its docs recommend validating the response against that same schema and using a low temperature for more predictable answers.

Then the grader fills in the sheet. Copy these columns:

#	Input (or a link to it)	Local answer	Local grade	Cloud answer	Cloud grade	Grader note
1	Voicemail 10/14 7:52 pm	name, address, "no heat", urgent	Fine	same	Fine
2	Voicemail 10/14 9:10 pm	address missing unit number	Fixable	full address	Fine	unit numbers
3	Email thread, warranty question	wrong job referenced	Wrong	right job	Fine	needs context

The rows above are an illustration of the layout, not results.

Grade with three words only:

• Fine: usable as is.
• Fixable: right idea, a person would fix it in under a minute.
• Wrong: misleading, missing the point, or invented.

Grade the local and cloud answers side by side but without knowing which is which if you can. Have someone shuffle the columns.

The wrong address rule

This is the part most tests miss. A reply can pass every format check and still be wrong. The JSON has an address field, the address field has an address, and it is the wrong house. Validation checks shape, not truth.

So add one column to the sheet: "Would a customer see this?" For every job where the answer is yes, the decision is already made. A person reviews before it goes out, no matter how many rows came back Fine. The test tells you which model drafts; it does not remove the review.

Decide the line

Count the grades for each model and use this rule:

1. Local is enough when nearly every row is Fine, the Fixable rows are quick, and the Wrong rows are the kind your validation or your reviewer would catch. Run that job locally.
2. Split the job when local handles the normal cases and fails on a clear pattern (long threads, warranty questions, anything with three problems in one message). Send that pattern to the cloud model or a person, and keep the rest local. Our hybrid AI guide covers how to route those cases.
3. Keep it in the cloud or with a person when the Wrong rows cost money: pricing, refunds, promises.

Write the decision at the top of the sheet in one sentence, with the date and the model names.

What to watch out for

• Fifty easy examples. If every input is clean, the test proves nothing. Keep the messy ones.
• The wrong grader. Someone who does not do the job cannot tell Fixable from Wrong.
• Queues. Ollama's FAQ notes that when memory is short, queued requests are processed in order. An overnight summary can wait; a reply to a caller cannot.
• Starting with five jobs. The Federal Reserve's 2026 small business report found 37% of firms planning to start AI named time to implement and train employees as a challenge. One job and one grader keeps the test small enough to finish.

How to tell it worked

You have a sheet with fifty graded rows per model, a one-sentence decision, and a reviewer named for anything a customer sees. Keep the sheet. When a new model comes out, rerun the same fifty and compare. That rerun takes an afternoon, and it replaces an argument with a number you made yourself.

Cortex sorts work the same way: plain rules first, private local models for routine work, cloud models when a task needs them, as described on our private AI page. If you would rather have someone walk your jobs with you, that is part of the free AI Operations Audit.