Done-for-you evals for AI agents in production

Find your agent’s failures
before your customers do.

We turn your SOPs, real cases, and your agent’s production responses into tests it has to pass. Our Agent Training Factory iterates prompts, tools, and models until it does. You change anything, and you know what broke before a customer does.

See how it works

A 15-minute call with Nik, the founder, then a written report on where your agent fails. No deck, no obligation.

Production AI systems shipped since 2022. Our insurance-claims agent work was written up by LlamaIndex.

We test against every major provider

OpenAIAnthropicGemini

The problem

Your agent mostly follows the rules.
“Mostly” is the problem.

It passed your last manual check. Since then you have changed a prompt, a tool, and the provider has updated the model. Three different bills are quietly accruing:

It drifts off your SOPs

Your policy says escalate before promising a refund. Usually it does. The time it did not is in a transcript nobody read, and that customer now has a screenshot. Rules that only live in a prompt are followed until they are not, and nothing tells you when that changes.

Nobody wants to touch the prompt

Every edit risks breaking three things that currently work. Nothing measures which three. So review threads fill with “did anyone test this?”, the prompt calcifies, and known issues ship for another quarter because fixing them is scarier than living with them.

Your model has a shelf life

Providers deprecate, reprice, and silently update models. When yours goes, migrating means re-testing every workflow by hand. So teams either burn weeks doing that, or keep paying old-model prices for old-model quality because switching feels dangerous.

Spot-checking transcripts catches the failures you happen to read. The ones that cost accounts are in the transcripts you don’t.

What we do

Evals score. Engineers iterate.
Your agent passes.

Not a dashboard, and not a spec document. The Agent Training Factory: a runnable suite built from your actual rules and cases, and a senior team that reworks your agent against it until the numbers hold.

01

Your bar, written down

We turn your SOPs, tone rules, real conversations, and your agent's production responses into scored cases: what pass, fail, and unacceptable look like for each workflow. Scoring fits the task. Deterministic checks where rules are hard, reference answers where they exist, model judges where nuance is needed, human review where judgment is.

02

Baseline, measured

Your agent runs the full suite. You see pass rates per workflow and the failures ranked by what they would cost you. For most teams this is the first real number they have ever had on agent quality. Everything runs in your accounts; your data and prompts stay on your infrastructure.

03

Iterate until it holds

Prompts, tool definitions, retrieval, guardrails, escalation routing, fallbacks. Many passes, and every pass re-runs the suite. A fix that quietly breaks something else gets caught in the next run, not in production. You see the numbers move mid-engagement, not at a final reveal.

Model freedom

Stop being locked
to one model vendor.

Once the suite exists, the model is a dependency you swap, not a rewrite you budget for. Run it against a cheaper model, read the diff, and ship if the numbers hold: the same quality bar at lower running costs, which is margin you keep every month. That includes open-source models, priced at a fraction of frontier rates and improving fast; the suite tells you the day one is good enough for your workload. And when a provider retires or reprices your model, the migration is an afternoon of eval runs, not a quarter of manual re-testing.

  • Lower running costs at the same quality bar
  • Open-source models become a real option
  • Migrate on your schedule, not the provider’s

Deliverables

What you walk away with.

The eval suite

Built from your real workflows, rules, and SOPs. Runnable, in your repo, yours.

A measured baseline

Error analysis included: what fails, how often, and which failures actually matter.

The iterated agent

Prompts, tools, and guardrails reworked until the numbers hold.

Regression runs in CI

Every future change is measured before release, automatically.

Monitoring in production

Quality, failed runs, latency, and cost. Silent failures stop being silent.

Docs and handover

Full ownership: repo, suite, dashboards, accounts. Your team runs it without us.

And when a buyer’s security review asks how you test the AI, you send artifacts, not a paragraph.

The other ways to get this

All reasonable.
All slower than they look.

Build it in-house

Completely doable, and after handover that is exactly who runs it. The cost is the month your best engineer spends discovering what to measure, how to score judgment calls, and which failures deserve a guardrail versus a human in the loop, while the roadmap waits. We have paid that tuition already.

Buy an eval platform

Good tools, and we work with them rather than against them. But a platform is homework: you still write the cases, define the scoring, and run the iteration loop yourself. The subscription starts the day you sign; the suite exists whenever you get around to building it.

Keep spot-checking

Free, familiar, and honestly fine at ten conversations a day. Past that, you are sampling. The failure modes that end accounts are rare paths on real customer data, which is precisely what casual reading of transcripts does not cover.

Where teams take it next

The suite is the start.
Four things it makes possible.

Ongoing eval coverage

Products move, and a suite from last quarter tests last quarter’s product. We stay on as the team that keeps coverage current, extends it to new workflows, and works the next weak spot. Pause or stop anytime.

Model migrations

A retirement notice or a price change worth acting on. We run the swap end to end: baseline, migrate, iterate to parity or better, with the diffs to prove it.

Hardening and red-teaming

Adversarial cases for the paths that worry you: prompt injection containment, tool misuse, data leakage. The abuse cases join the suite, so they stay tested on every change instead of being a one-off report.

The next agent

If the agent does not exist yet, we design and build it the same way we test them: evals first, then the product. The portfolio below is what that looks like shipped.

Proof

Shipped. Still running.

Production AI systems we built for founders, including the insurance-claims agent LlamaIndex wrote up.

Insurance Claim Agent

Insurance Claim Agent

Autonomous medical insurance claim analysis and estimation

45-minute manual review → 7 seconds

  • Insurance
  • Healthcare
  • Automation

Read the LlamaIndex case study

MemberHub

MemberHub

AI search across community members

5,000+ members searchable in natural language

  • Community
  • Search
  • AI

Visit memberhub.ai

Nik Volynkin, founder of Scale Factory

Who you talk to

No account managers in the loop.

We built an AI product in the GPT-3.5 era, grew it to real users, and sold it. Some of the models it ran on no longer exist. So I know the hard part was never getting the agent to work once. It was knowing it still worked after the next prompt change, the next tool, the next model release. That is the problem Scale Factory removes. You talk to me before we start, and you stay close to the senior engineers doing the work.

“The goal is not a perfect agent. It is a bar you can measure, failures you can see, and a product you can change without holding your breath.”

Questions engineers actually ask

What needs to exist before you start?

An agent doing real work, live or close to it, and someone who can walk us through the rules it must follow. Your SOPs, real cases, and your agent's production responses are the raw material for the suite. If the agent is not built yet, see “The next agent” above; that is a different starting point and we take it.

We have engineers. Why not build this in-house?

You will run it in-house: the engagement ends with handover, not a dependency. What you buy is the finished harness and the iteration cycles without the month of working out what to measure, how to score it, and which failures deserve a guardrail versus a human in the loop.

What does “follows our SOPs” look like as a test?

Each rule becomes cases that probe it. “Escalate refund requests above the limit” becomes clean cases, edge cases, and adversarial phrasings, scored automatically where rules are hard and by judge or human where nuance is needed. You get per-rule pass rates, so “does it follow policy” stops being one anxious question and becomes a list of numbers.

Model judges grading model outputs. Isn’t that circular?

Unchecked, yes. That is why judges are only one scoring tier, they are calibrated against human-labeled cases before we trust them, and anything they flag as borderline routes to human review. Where a deterministic check can do the job, it wins.

What if the baseline is worse than we thought?

It often is, and that is the useful part. The baseline turns a vague worry into a ranked list of specific failures, and then you watch the number climb against a fixed bar. Nothing about the baseline leaves your accounts.

How much of our team’s time does this take?

A few hours total from someone who knows the rules and can grant access, plus a short weekly review. We do the building. If you have an engineer who wants to learn the harness as it grows, we work in the open in your repo.

Which stacks and models do you work with?

Every major provider and the common agent stacks. The suite is provider-agnostic on purpose. That is exactly what makes switching models a decision instead of a project.

Does this cover prompt injection?

We test for it and contain it: guardrails, tool permissions, escalation on suspicious input. Nobody can honestly promise prevention, so we do not. The adversarial cases stay in your suite, so every future change gets re-tested against them.

The agent works. Make that provable.

Bring the mission-critical agent whose responses you care about.

Fifteen minutes with Nik. Walk him through what the agent does, then send a sample of real conversations. You get back a short written report: the failure modes we found in your transcripts, ranked by what they would cost you, and what we would fix first. It is yours whether we ever work together or not.

  • Founder call, no sales deck
  • A written failure report, free
  • No obligation