We Audit AI Visibility for a Living. So We Audited Ourselves.

Can you trust the score?

Before anyone buys an AI visibility audit, they are really asking three quieter questions. Can I trust the score this tool gives me? Will the number be the same tomorrow? Is this a serious company, or a wrapper around someone else's model?

Here is our answer, up front. AI models change silently. A provider can update a model and the same prompt starts scoring differently, with your code unchanged and nobody warned. If we did not watch for that, our scores could drift and no one would notice. So we test our own scoring every week, against a fixed set of domains, and we refuse to let the system quietly correct itself. Stability is not something we claim. It is something we measure, on the record.

We can say that with a straight face for one reason. We sell AI visibility audits, so we turned the same lens on our own stack first. This is the full report, including the parts that were not flattering.

Will your score be the same tomorrow? We check every week.

Here is the failure mode that no number of unit tests can catch. A provider silently updates a model, and the same audit prompt starts scoring differently. The code is unchanged, the tests are green, and the product is quietly broken.

For a company whose product promise is a score you can defend, silent score drift is the single most on-brand failure possible. So we built the smallest thing that could notice it, and we called it eval-lite.

Eval-lite is 12 fixed golden domains of mixed difficulty, including a never-changes low anchor and real customer-profile sites. Every Monday they are re-scored through the exact same code path a real audit uses, results are stored, and an alert email fires only when a score leaves its expected band. Silence means healthy.

It earned its keep before it even shipped. Baseline sampling showed one domain scoring 59, then 59, then 45 on identical code, pre-existing nondeterminism we would never have seen without the harness. It also verified the reassuring flip side: our own domain and several customer-profile domains scored identically across dozens of historical runs, a standard deviation of zero. Stability is not claimed here. It is measured weekly, on the record.

One design choice matters most. Eval-lite is deliberately not self-correcting. It detects, humans decide. A scoring system that quietly recalibrates itself to make an alert go away hides its own instability, which is the exact opposite of defensible.

The foundations that make a stable score possible

A weekly test tells you whether the number moved. It does not, on its own, make the number trustworthy. That comes from the architecture underneath: how our code talks to models, how changes reach production, and whether the whole system is accountable to us. So we audited that too, and it is where the unflattering findings live.

We measured before we defended ourselves

The first move in any honest audit is an inventory, not an opinion. We gave our coding agent one read-only search prompt: list every file that talks to an AI provider directly.

The honest result was mixed. A shared multi-provider hub already existed, and for most providers it was working: Anthropic, Gemini, and Grok were 100 percent centralized, with Perplexity down to a single stray call. But OpenAI usage had quietly decayed into 23 files holding their own private clients, spread across 11 API routes, 9 library modules, and 3 scripts. We rated the exposure a 7 out of 10, roughly a 40 to 60 split of hub versus inline calls.

The lesson before the lesson was the uncomfortable one. A hub existed, and drift happened anyway. This is the whole story of AI technical debt in miniature: good intentions and a clean abstraction are not enough on their own. Without enforcement, abstractions rot. We have written before about how AI-generated code accrues technical debt faster than teams expect, and here it was in our own repository.

A score is only as trustworthy as the plumbing beneath it, and ours had quietly drifted. Counting that drift was the first step to making the number defensible.

Phase 1: one hub, mechanically enforced

Consolidation took one afternoon and changed zero behavior. We rerouted every provider call through a single hub file: 20 call sites plus 2 scripts migrated, and the stray Perplexity fetch fixed along the way.

The migration is not the interesting part. The lock-in is. We added ESLint rules that mechanically ban provider SDK imports and provider hostnames anywhere outside the hub, canary-tested so we knew they actually fire, plus a hard rule written into our agent instructions file so both humans and AI agents get stopped at the door. This is the difference between hoping the abstraction holds and guaranteeing the category of drift becomes impossible. It is the same discipline we apply when we prevent AI technical debt with structured engineering rather than cleaning it up after the fact.

We held ourselves to one constraint: no prompt, model, temperature, or behavior changes. Client plumbing only. All 926 tests stayed green. Where the refactor did introduce a small delta, a timeout ceiling on one long-running call and a lazy API-key failure, we disclosed both explicitly in the pull request rather than letting them hide inside a large diff. Small, intentional, documented.

One enforced path for every model call is what lets us swap or re-test a model deliberately, in one place, which is exactly what a stable score requires.

Phase 2: a model registry and full observability

With everything flowing through one hub, two upgrades became one-file changes instead of codebase-wide surgery.

First, a task-keyed model registry. Features now name a task, such as chat, advisor, or tool analysis, and one config file maps those tasks to models. Model swaps and A/B tests become a one-line change, which matters more than it sounds: changing a model on purpose, in one place, is the opposite of a model changing under you by surprise.

Second, per-call observability. Every AI call now writes exact tokens, latency, and status to a dedicated table. Nothing our system does is invisible to us. If a call slows down, errors, or starts behaving oddly, we see it in the data instead of guessing.

One detail matters more than the plumbing. The agent building this refused parts of the spec that would have weakened integrity. Our probe models cannot be overridden by an environment variable, because our public disclosure of which models we measure must match what we actually query. Good constraints outrank convenient flexibility, and we would rather ship the honest version.

When every call is named by task and logged with exact tokens, latency, and status, nothing our scoring does is invisible to us, and an invisible system cannot be a defensible one.

The database pass: 79 warnings to zero

While we were in the machine room, we ran Supabase's own database advisors against our project. They flagged 79 row-level-security performance warnings: policies re-evaluating auth functions once per row, which turns into quadratic pain as user counts grow, plus 50 stacked-policy redundancies.

The fix was logically identical policy rewrites that evaluate once per query instead of once per row, applied as three verified migrations. Seventy-nine to zero. We did it while the affected tables were nearly empty, which means we hardened the foundation before load rather than under it. Timing is part of the craft.

There was a bonus finding, and it was not flattering. Six tables in production had no migration provenance at all. They had been created by hand before our numbered migration history existed. They are now documented, and the migration scripts are hardened to be replay-safe, so the repository and the database finally agree. The automated review bot's findings on this pass were engaged honestly too: we accepted what was right and pushed back with reasoning where it pattern-matched wrong. How we handle that disagreement is its own section.

A scoring system sitting on a database the repository cannot fully account for is not defensible. Now the repository and the database agree, and the number rests on ground we can explain.

How does none of this ship on one model's say-so?

None of it did. Every change in this post shipped as a reviewed pull request, and the builder and the reviewer are deliberately different models.

Here is the pipeline. Our coding agent (Claude Code) builds the change. A second, independent model (OpenAI Codex) reviews the diff against a written security-and-business-risk charter. An automated review bot runs in parallel. A human approves before anything touches the main branch.

Why two models and not one? A reviewer from a different training lineage catches what the builder is blind to. It is scoped to the diff, not the whole repository, and it ranks findings by real risk, security, auth, data leakage, and regressions, rather than style nits. Two models disagreeing productively is worth more than one model agreeing with itself.

The point is not that the machines are always right. It is that a flag gets a reasoned decision on the record, not a reflexive patch and not a reflexive dismissal. The clearest example was the rate limiter. The review bot flagged our rate-limiting logic. We disagreed with the specific fix it proposed and declined it, with the reasoning written out in the pull request. But the flag pointed at a real gap for a different reason, so we added a minimum-interval guard on our own terms and documented the whole accept, decline, and why in the same thread. The bot did not get its way. It got us to a better place than either the bot or a reflexive dismissal would have.

The guardrails apply to the reviewer too. It never reads secrets or environment files, referencing variable names only. It never quietly rewrites payment, auth, or gate logic. It never churns dependencies. The same restraint discipline runs through the entire body of work.

The safety harness around the score

A stable score also has to fail safe when something goes wrong, so the system that produces it runs inside a few hard guardrails.

Every AI call has a hard time limit, so one slow or stuck model call cannot stall a report or leave you watching a spinner. A single switch can turn every AI feature off at once, so if a provider starts misbehaving we can pull the AI instantly instead of serving you a broken result. No AI tool in our pipeline, including our own reviewer, is allowed to read secrets, credentials, or environment files. And raw HTML injection is banned outright and fails our build automatically, while every piece of text a person submits is cleaned before it is used, so untrusted input cannot turn into an attack on you or your data.

None of these are features you will ever notice, and that is the point. The number rests on a system built to fail safe, so a bad model, a slow call, or a hostile input degrades quietly instead of corrupting the result you rely on.

What we deliberately did not do

Restraint is part of the credibility, so here is what we left alone on purpose.

We did not bolt on a third-party evaluation or observability platform. We already had most of what we needed in-house, so we built the missing piece in a day on our own stack. We did not smuggle behavior changes into refactors: every phase either changed zero behavior or disclosed its deltas explicitly. And we deferred several things with reasons, including unused-index cleanup, a module split we did not need yet, and cosmetic policy canonicalization on tables that did not warrant it. Knowing what not to fix is half the discipline.

The scoreboard: before and after

Every claim in this post reduces to a before and an after, and the ones that matter most sit at the top: the score is stable, the drift is watched, and the system is accountable to us.

What this means for your stack

If your product is a number, you owe your customers a system that notices when that number drifts, plus the discipline not to let the system quietly fix it. That is the whole reason this post exists, and it is the first thing to build, not the last.

Three more takeaways travel to any AI-heavy stack, not just ours.

Abstractions decay without enforcement. Our hub existed before the audit, and drift happened anyway. Lint rules plus agent instructions are what made the fix stick. Measure before you defend yourself: the inventory prompt took two minutes and replaced opinion with a number. And behavior-preserving refactors plus honest delta disclosure equal changes you can ship without moving the number you promised to keep stable.

The principle underneath all of it is simple. Clean abstractions do not just help humans. They made every subsequent AI-agent task in this story faster and safer, because an agent that only has to change one file does not have to read the whole repository to do it. This is the level of rigor we bring to our own site. It is the same rigor we bring to auditing yours.

Conclusion

We did not write this to look flawless. We wrote it to show the work behind a single promise: the score we give you is stable and defensible. We test it weekly, we refuse to let it secretly correct itself, and underneath it we consolidated 23 drifted files, cleared 79 database warnings, documented six untracked tables, and grew our tests from 926 to 1,109, each fix making the number harder to break. The receipts are the point. If this is the rigor we hold ourselves to when nobody is watching, it is the floor for what we bring to your stack.