Data obfuscated for a secure public portfolio
Similarity Is Not CorrectnessAgentic AI retrieval governance
Semantically similar is not the same as commercially correct. An agent can retrieve something close enough to sound right and wrong enough to cost money, then act on it before anyone looks.
The obvious fix (a governance gate between retrieval and action) fails the same way retrieval does: the gate is itself a retrieval that can go stale, mis-scoped, or drifted. This case tests that fix until it strains, then resolves it.
Siarhei Mardovich designed an agentic-AI retrieval governance case: a failure taxonomy, 4 interactive simulations, and the Action Ledger decision instrument, arguing for calibrated, reversible, self-auditing autonomy instead of stacked gates. Domain: AI trust, retrieval risk, model-risk governance. Role: Principal Product Designer and prototype engineer.
Role, approach, and outcome
Role
AI-augmented Principal Product Designer and prototype engineer. Authored the analysis, the simulations, and the instrument end to end.
Approach
State the problem honestly, test the easy fix against its own tensions, then design the resolution and make it operable rather than rhetorical.
Outcome
Decision reached: do not stack gates; calibrate one reversible, self-auditing boundary instead (Fig 5–7). The Action Ledger ships that boundary as an auditable record-keeping instrument. No production deployment, no behavioral outcome is claimed.
Retrieval is the silent failure surface
In search, a person reads the result and decides what to do. In agentic systems, the agent retrieves, reasons, decides, and executes. The validation step disappears, so a retrieval that is semantically close but commercially wrong does not surface as an error. It surfaces as a confident, executable answer.
4 failure modes account for most of it: attribute ambiguity, variant confusion, stale knowledge, and policy drift. The standard fix is a governance layer between retrieval and action: freshness gates, attribute validators, constraint filters, confidence scoring, audit trails, graceful degradation. Measure operational correctness, not ranking quality.
The sharpest version of the problem: the next generation of critical AI failures may be retrieval failures, not model failures. Silent, confident, wrong. So gate the retrieval before the agent acts on it.
The gate is itself a retrieval that can be wrong
A freshness gate, a constraint filter, an attribute validator: each is a rule that must itself be retrieved, versioned, and applied. A stale threshold, a mis-scoped constraint, a drifted policy in the governance layer fails exactly the way retrieval fails. Adding gates does not remove the silent-failure surface. It adds another one, a layer up.
2 more tensions sit inside the easy fix. Every gate that escalates pushes work onto people, and human review has a finite throughput. Past that ceiling the queue backs up and reviewers rubber-stamp, so governance manufactures the overconfidence it set out to remove. And a system tuned only to avoid acting wrongly will increasingly fail to act when it should: errors of commission fall, errors of omission rise.
Backlog: 0 queued
Review quality: 0.95
So the easy fix is not free, not infallible, and not one-sided: gates are fallible retrievals, escalation has a ceiling, and over-gating trades errors of commission for errors of omission.
Calibrate one decision boundary
Keep what holds: retrieval is a real failure surface, and governance belongs between retrieval and action. Answer what does not: govern by a calibrated operating point, not a wall of gates. The right point is where the sum of acting wrongly and failing to act is lowest, chosen on purpose and moved as conditions change.
3 moves follow. Treat autonomy as a tunable threshold and find its interior optimum. Gate by reversibility and stakes rather than uniformly: block hard only where an action is both high-stakes and hard to undo, and prefer act-with-cheap-undo elsewhere. Make the governance layer self-auditing, so the recursion in Figure 3 is closed rather than ignored. The loss surface shows why the operating point has to move with how irreversible the action is.
I chose calibration over stacked gates because N+1 gates do not reduce the silent-failure surface. They multiply it (see Fig 3).
Figure 5. Autonomy threshold
Omission: 0
Net loss: 0
Figure 6. Reversibility × stakes
Figure 7. Loss surface
In one line: do not stack gates. Calibrate one reversible, self-auditing decision boundary, and move it by stakes and irreversibility. The instrument below is that idea, made operable.
Action Ledger
Decision PAY-48291: how the ledger surfaces a wrong action
The ledger is not a dashboard of outcomes. It is a decision surface: every row shows where retrieval and action diverge, and the interface makes the stakes, reversibility, and gate logic explicit, whether the action was blocked or, as in this case, already executed.
0.94. High enough to surface first, wrong enough to price the wrong tier.
Where this transfers
The transferable pattern is the resolution: calibrate one reversible decision boundary and govern the governance. It applies anywhere an agent executes on retrieved knowledge. The same interaction design discipline (information architecture for trust, decision-support UX, and auditable interaction design) scales across AI governance, RegTech, model-risk management, compliance operations, and fintech product design roles.
| Domain | The irreversible, high-stakes action that earns a hard gate |
|---|---|
| Payments and Spend | Same-day ACH release on the wrong pricing tier |
| KYC/KYB | Onboarding clear on a stale ownership record |
| Trade Surveillance | Alert dismissal under a drifted rule version |
| Sanctions Screening | Clearance against an out-of-date list |
| Credit and AML | Limit or disposition set on a confused variant |
What I would ship differently
The next production step is calibration: replace the illustrative cost curves, queue model, and loss surface with the organization’s own baseline error rates, reviewer throughput, and reversal costs.
The design principle stays the same: high-similarity retrievals should not become silent actions. They need a ledger that exposes confidence, reversibility, policy freshness, and the cost of a wrong step before the agent acts.
The assumption I brought in was that governance means more gates. Figure 3 broke it: the gate is a retrieval too. The rule I carry forward: govern the governance before adding another layer.
Technical details
The case page is static, crawlable HTML and CSS in the siarheimardovich.com design system. The interactive figures use inline SVG. The Action Ledger instrument is a single-file React 18 decision workspace with a Material 3-informed dark-glass system, embedded by iframe with a descriptive title and lazy loading.
Static HTML and CSS, no build step, semantic structure, single h1, figure/figcaption wrappers, WCAG AA-aligned contrast and focus states.
Hand-authored inline SVG diagrams for the pipeline, recursive governance, scatter plot, queue model, autonomy frontier, reversibility matrix, and loss surface.
Single-file React 18 decision workspace with Material 3-informed dark-glass components, embedded by iframe with loading=”lazy” and a fallback screenshot.
ARIA labels on figures and iframe, keyboard-operable controls, prefers-reduced-motion honored, color not the only information channel.
Product design, interaction design, user experience, decision-support UX, information architecture, trust architecture, and model-risk-aware interface design.
Agentic AI, RAG governance, retrieval risk, model risk management, SR 11-7, MRM, RegTech, KYC/KYB, trade surveillance, sanctions screening, reversibility.
Let’s Talk
Open to principal/staff product design roles · Greater New York City Area · siarhei.mardovich@gmail.com · LinkedIn · Resume