LLM Cloud-Use Audit

How much of your LLM bill can come home?

A fixed-scope audit that measures how much of your AI workload can run on your own hardware at the quality you require, and shows you the evidence.

Most apps send every request to the cloud. That costs money, sends your data out of the building, and ties you to someone else's uptime. Local models have become good enough for a lot of everyday work. Nobody can tell you how much of your work, though, until it is measured against your own requests.

How the audit works

1 — Capture

Your app points at a small proxy running on your own machines, in log-only mode. Your traffic still goes to your current provider. Only a web address changes.

2 — Measure

Your real requests become a test set with reference answers. Local and lower-cost models are run against it on your hardware, graded by a written rubric, and spot-checked by hand.

3 — Decide

You get the trade-off curve and a recommended setting: how much work can stay local, what it saves, and what it costs in quality and speed. You can stop there.

Optional — Follow on

I deploy and tune the dispatcher for you, and re-run the measurements as models improve, because last quarter's numbers go stale quickly.

What the report tells you

  • Cloud dependence

    The share of your requests that truly need the cloud to meet your quality bar. This is the headline number, and the goal is to bring it down.

  • Cost per 1,000 requests

    Today's cloud cost against the local alternative, including electricity, so the comparison is honest.

  • Quality against a reference

    Each answer is scored against a strong model's reference answer, with the misses sorted by type, so you see what kind of mistakes you would be accepting.

  • Speed

    Time to first word and total time per route. Local is often slower per answer, and the report says by how much.

  • What still needs the cloud, and why

    The request types where local models fall short, with the reason for each. That list is your roadmap for reducing cloud use further.

  • What would leave your building

    Which sensitive categories show up in your traffic, how well they are caught before anything is sent, and exactly what a redacted request looks like.

Your data stays yours

It runs on your machines. Capture and measurement happen on your hardware. I do not hold your data, and the work is done under a non-disclosure agreement.

Sensitive content stays local. Categories you mark as sensitive are pinned to your own models whatever the settings. Redaction reduces risk for everything else, but it is not a guarantee, and I will not describe it as one.

Cloud calls are capped and logged. Reference answers from a cloud model are generated only with your approval, from redacted requests, under a hard spending cap that fails closed, with a log of exactly what was sent.

Early evidence, from my own pilot

I ran 20 everyday chat questions, from unit conversions to a probability puzzle, through a local model and three Claude models. This is a small test on my own setup, and it shows the shape of the answer, not a promise for your workload.

Score out of 20, and the cost of running the 20 questions
Model Where it runs Score Cost for 20
Gemma 4 26BLocal, on a Mac mini19.5$0
Claude Haiku 5.5Cloud19.5$0.005
Claude Sonnet 5.5Cloud19$0.063
Claude Opus 5.5Cloud19$0.21

The best local model matched the cloud models on this set. Its one slip was arithmetic: it computed one multiplication a few cents off. Smaller local models failed on the probability and time-zone problems.

Cost for 1,000 short chats: about $0.03 of electricity locally, against $0.23 on Haiku, $3.16 on Sonnet and $10.52 on Opus. The price of staying local was time: 6.1 seconds per answer against 2.5.

Read it with care. Twenty questions, one grader, so scores are good to about 5%. Prompts were short, which understates how cloud cost grows with long conversations. Your own requests may behave differently, and measuring that is what the audit is for.

Who it is for

Teams with a real API bill. You spend enough on LLM calls to notice, and have no time to evaluate local models properly.

Privacy-sensitive practices. Law, accounting, clinics, small consultancies: places where client material should not be casually sent to a third party.

Shops with Apple silicon. You already own Macs that could host local models, and want to know which models, and for which work.

How an engagement works

Scope
Fixed, written down before we start
Duration
About 2 weeks
Fee
$2,500 fixed fee
You receive
A written report, the test set, and a runbook

About

I am Rob Roy, an independent software engineer based in Maine, with more than forty years in the field. Much of it was release engineering and configuration management at SGI, Informix and IBM: work that depends on measuring things carefully and being able to reproduce the result. I now build privacy-first iOS and macOS apps, with no accounts and on-device processing, under Robroy Online LLC. This audit applies the same habits to a new question.

Want to know your number?

Tell me roughly what your LLM usage looks like and which data you worry about. The first conversation is a no-cost 15-minute call; I will tell you whether an audit makes sense, and what it would cover.

Email: info@robroy.online