What is Rhondda? AI system design is a cascade of small, interdependent design choices and compounded effects. Tweak one parameter, and something breaks three layers down. Add agentic AI, and that complexity explodes. Yet, no one has systematically mapped how these choices actually interact into a usable, causal model for designers.
Rhondda is an empirical research program that maps the impact of design choices on performance and reliability across all LLM-based systems. It is based on the idea that building reliable AI through trial-and-error is not enough: we need controlled, comparable measurements. That would give designers the evidence they need to avoid costly architectural misconfigurations before they bake into production.
Rhondda is not another benchmark, not a leaderboard, not a best-practice guide. It is a registry of qualified effects.
Its name comes from the Welsh valley where Archie Cochrane conducted the field surveys that laid the groundwork for Evidence‑Based Medicine. The name is a reminder that rigorous, cumulative measurement starts in the field — not in theory.
A High-level view of Rhondda Rhondda is designed to produce, qualify, version, and maintain a registry of comparable measurements. For any given task, it maps how system-design choices affect both task performance and reliability in LLM-based AI systems.
We establish these effects through controlled comparisons. A change in performance is only attributed to a specific factor if that factor has been strictly isolated, meaning compared configurations differ only by that single variable on the same task. This isolation discipline is the foundation of the program: without it, the registry would produce a mere accumulation of scores. By enforcing this control, Rhondda creates a composable registry, allowing designers to analyze not only individual effects but also how these variables combine and interact in larger architectures.
To make these findings actionable, this registry feeds Atlas, a free public interactive cartography designed to make the empirical results navigable and diagnostic for system designers. The qualified data is intended to remain publicly and durably accessible, subject to legal, security, privacy, data rights, and governance constraints.
This downstream use dictates the program's primary criterion: the data must be load-bearing, meaning it is characterized well enough for an AI system designer to rely on it in real-world production decisions rather than simply being publishable. The registry’s qualification standard ensures we produce measurements to build upon, not just measurements to defend.
In this sense, Rhondda is use-inspired research as much as it is knowledge-inspired; actionability is achieved through an increase in methodological discipline, not its relaxation. We do not trade scientific rigor for actionability. Instead, we add the missing layer of characterization required to build upon that rigor. The existing literature is not wrong. Its findings are often rigorous – but they are not characterized well enough for a designer to act on.
What "Research Program" Means The term research program carries three coexisting definitions in Rhondda:
Empirical Accumulation: A coherent set of studies sharing a common methodology, designed to accumulate over time. The measurements are not isolated experiments; they aggregate into a composable substrate whose value grows with its coverage.A Methodological Framework: A "hard core" of non-negotiable assumptions (such as the isolation discipline) and a "growth heuristic" that dictates what deserves to be measured and what is excluded.The Operational Unit: The organization itself, comprising the protocols, the experimental infrastructure, the methodological expertise, and the pace of coverage.Additionally, Rhondda acts as a public guide. By mapping the covered perimeter, the explored zones , and high-value methodological gaps, it makes the program's trajectory readable and helps orient distributed research worldwide.
Because no single team can cover the entire design space, this framework allows third-party researchers to contribute measurements that easily integrate into the registry.
LLM-Based Systems According to Rhondda An LLM-based system refers to a compound application in which a Large Language Model is merely one component, rather than the model alone. The system includes deterministic code, retrieval mechanisms ( RAG ), external tools, databases, validators, human interfaces, orchestration policies, and guardrails .
This definition explicitly spans non-agentic configurations, such as single-call prompts, Chain-of-Thought , and simple RAG, as well as agentic configurations, including tool-calling agents , ReAct loops , and multi-agent setups.
This wide scope distinguishes Rhondda from most evaluation literature, which focuses either strictly on models or exclusively on complex agents. In this program, the decision to agentify a workflow is not a starting assumption. It is simply one variable to be empirically tested and measured against simpler, non-autonomous alternatives.
What we measure, and how The Unit of Observation: The Task In Rhondda, every measurement we take is strictly tied to a specific task. The task acts as the absolute foundation of comparability—you cannot directly compare an effect measured on a math problem with one taken on a summarization tool without an explicit framework.
The Axis of Variation: System Design and Design Factors System design covers every structural choice you make when building an AI application. These choices range from massive architectural decisions (like using an agentic loop or a RAG pipeline) down to the smallest localized tweaks (like rewording a system prompt or reordering the tools).
To measure these choices, we isolate them into design factors . A design factor is simply a single variable we can control. It might be a simple toggle (like turning a feature on or off) or a sliding scale (like tweaking a temperature parameter).
If we want all these independent measurements to stack up into a useful registry over time, they must share a rigid reference point. That is the role of our system design grid:
What it is: A pre-empirical map that outlines every possible design choice before any measurement even happens.What it does: It asserts that a lever exists and can be pulled, but it makes no assumptions about what pulling it will actually do. The effect itself must be proven with data.This creates a necessary bridge in how we talk about these things. In the experimental setup, Rhondda measures "design factors." But out in production, downstream tools like Atlas help you navigate those exact same levers as " system design choices ." They are the exact same objects, just viewed from two different angles: one for measuring, one for building.
The Experimental Object: The Configuration If the design factor is the variable we change, the configuration is the actual object we test. It is a fully built system, running a specific model, on a specific task, under strict, documented conditions.
Things like how autonomous the system is, or whether a human is reviewing the outputs, can act as either variables we are actively testing, or constraints we are forced to work within. We record this status explicitly so we never mistakenly attribute an effect to the wrong cause.
What We Actually Measure: Core Properties When comparing configurations, Rhondda strictly evaluates two distinct, non-substitutable core properties:
Task performance : Does the system get the job done right? (Scored using local, task-specific criteria). Reliability : How predictably does the system behave? (Evaluated across 4 universal dimensions). Finally, execution costs (like latency, token usage, or API calls ) are not treated as core properties to measure, but they are mandatory metadata. They provide the exact numbers designers need to weigh performance gains against operational costs.
The Derived Output: The Measured Effect In Rhondda, we do not just report raw numbers; we measure effects. An effect is the observed difference in a core property (either task performance or reliability) between comparable measures.
Because an effect never exists in a vacuum, we rely on controlled comparisons to isolate what truly causes a change in outcome.
The Task in Rhondda In Rhondda, the task is never a neutral backdrop. It plays two absolute roles: it is the unit of observation (the exact grain at which an effect is attributed) and the condition of comparability . You cannot blindly compare an effect measured on a math problem with one taken on a summarization tool. Depending on the task, a design factor might not just change in intensity. Its effect might reverse entirely.
The Task is Not the Prompt A common mistake in AI evaluation is confusing the task itself with how it is presented to the model. Rhondda enforces a strict firewall here. A task is defined by three absolute components:
Inputs: The raw data you give the model.Expected Outcome: What the model is supposed to produce.Success criteria: The definitional intention of how we judge if it succeeded.Only the first two are exposed to "presentation levers" (e.g., rewording the instruction or asking for JSON). Success criteria have no presentation lever. This means that rewriting a prompt does not change the task. It only changes its presentation. The task only changes if the success criteria change.
The Triple Invariance of a Task
A task remains exactly the same across three levels of changes:
Invariant under Presentation: Changing the prompt from "summarize this" to "extract three points" is the same task under a different presentation, unless getting exactly three points becomes a new success criterion.Invariant under Metric: The metric simply scores the criterion. Changing the metric does not change the task.Invariant under Evaluator: Swapping a human judge for an LLM-as-judge does not change the task. Both the metric and the evaluator sit below the task definition. Item vs. Task-Type To read data correctly, Rhondda strictly separates the two granularities of a task:
Level What it is Its role in Rhondda Item A single, unique instance of an input. The unit of scoring and statistical variance. Task-Type The abstract problem space and its criteria. The unit of attribution and generalization.
Confusing the two leads to fatal statistical errors, like mistaking the natural dispersion between test items for a variation between tasks.
Atomicity and Sub-Tasks A task can be atomic (handled in a single step) or composite (broken down into sub-tasks). Crucially, whether a problem is broken down is an orchestration choice (a design factor structured as a DAG or state machine), not an innate property of the task.
Rhondda imposes strict rules on how we measure these decomposed systems:
Well-formedness: To be measured as a sub-task, an intermediate step must be scorable. It must have its own inputs, outcome, and criteria. A silent reasoning step is not a sub-task.The Part-to-Whole Trap: You can measure an effect at the sub-task level, but you cannot mechanically extrapolate it to the global task. An improvement at Step 2 might be absorbed or completely negated by Step 3. Sub-task criteria are dictated by the cut, not derived from the global task.Dynamic Tracing: If the orchestration is dynamic (like an agentic loop), the structure varies per run. The level of analysis can no longer be declared upfront. It must be reconstructed from the execution trace. Criteria vs. Attributes Every task carries two entirely different sets of properties:
Property Nature Rule in Rhondda Performance Criteria Determines if the system succeeds. It carries the local identity of the task. Fiercely local. We measure every criterion separately. It is an error to canonize a universal list. Weighing tradeoffs is deferred to downstream tools like Compass. Task Attributes Meta-properties that dictate how the measure can be interpreted. Operates at the task level. Attributes (e.g., verifiability, scoring reference, requiring an LLM-as-judge) allow us to group tasks into formal Families
Because Performance Criteria define the task's identity, Rhondda enforces three strict rules for them:
The "If vs. How" Test: A trait is categorized by its role. Asking for a JSON format is a criterion if it counts towards success, but format consistency under perturbation is Robustness (a reliability dimension). Definitional Overlap vs. Correlation: A well-formed criterion must name a distinct aspect of success. Two criteria can statistically correlate (e.g., coverage and factuality) without being definitionally identical. They remain distinct criteria.Under-characterized Criteria: If a task requires a specific aspect of success, but no metric currently exists to score it, it remains a criterion . It is simply under-characterized, not abolished.Cross-Task Generalization Because an effect is bound to its task, asserting that a design choice works universally is a generalization hypothesis . We test it by applying the identical design factor across a characterized Family of tasks. But properties generalize asymmetrically:
Consistency & Safety : Defined identically across all tasks. Their cross-task comparison is straightforward. Robustness & Predictability : These import the local performance criteria to function (e.g., maintaining acceptable performance). Their comparison inherits the locality of the task. Task Performance : Generalizes only if the underlying intention of the criterion is shared across the family. Finally, generalization is bound by a strict product: Perimeter of Tasks × Perimeter of Models . The model limits the validity as much as the task. (Note: in Rhondda, to "interpolate" or "extrapolate" refers strictly to membership in a family, not mathematical distance, because the task space has no metric). If an effect varies inside a characterized family, the hypothesis is either rejected, or it proves the family is missing a hidden attribute that drives the variation, requiring the family to be re-characterized.
The problem of task noise Every measurement in Rhondda captures a shift in Task Performance or Reliability caused by a single Design Factor (lever).
However, if a task itself introduces variation (due to subjective criteria, ambiguous ground truth, or unstable grading), this task noise contaminates our data. We cannot isolate system design effects if the task itself is a source of undocumented variance.
To build a reliable registry, we need to determine which tasks introduce noise and classify them accordingly. This allows us to control the exact perimeter of tasks we evaluate depending on the measurement cycle we want to run.
The Twelve Task Attributes To make our measurements comparable and trace our perimeter, we catalog every task using twelve stable parameters (our task attributes). You can think of them as answers to three basic questions:
1. How is the task verified and scored?
Attribute Name Description AT-verifVerification Mode How success is checked: exact reference matching (such as Exact Match or EM ), executable code evaluation (such as execution-based evaluation ), or stochastically judged. AT-evalStochastic Evaluator Whether the grading instrument itself is stochastic (such as an LLM-as-judge ). AT-verdictVerdict Granularity The shape of the judgment (binary, ordinal scale, field-by-field, or structural matching). AT-criteresDecomposability Whether success is measured by a single metric or a multi-criteria bundle (such as factuality and formatting).
2. What does the response structure look like?
Attribute Name Description AT-sortieOutput Type The structural shape of the expected output (atomic, key-value fields, segments, executable query, or free text). AT-formatFormat Requirement Whether formatting rules are a constitutive success criterion (such as a Structured Output constraint), or merely accessorial. AT-espaceResponse Space Whether the range of possible answers is closed (like multiple choice) or open (like essays).
3. What is the task's complexity and context?
Attribute Name Description AT-profReasoning Depth The required inferential effort (lookup, single-step, or multi-step reasoning). AT-domaineDomain of Knowledge The subject matter (such as math, law, or medicine), used as a stratification variable. AT-tracesTrace Availability Whether intermediate reasoning steps are recorded and scorable. AT-sourceExternal Evidence Whether the task is self-contained or depends on retrieved external data (such as a RAG system).
The Baseline: The Discrete Verifiable Sub-Space By selecting the cleanest modalities of the attributes defined above, Rhondda isolates a specialized baseline environment called the Discrete Verifiable Sub-Space . Think of it as a laboratory cleanroom: we mathematically isolate a zone where task noise is completely eliminated. We define this sub-space as:
Discrete Verifiable ≡ Exact Verification (AT-verif) × Closed Response Space (AT-espace) × No Judge (AT-eval) × Atomic Output (AT-sortie)
This definition represents a strict intersection of coordinates:
Exact Verification (AT-verif): Checked deterministically against a known ground truth.Closed Response Space (AT-espace): Restricts the potential output to a pre-defined set of options.No Judge (AT-eval): Eliminates any stochastic variance introduced by an LLM-as-judge evaluator.Atomic Output (AT-sortie): Returns a single flat value or structured fields rather than a complex sequence.Because task noise is mathematically eliminated here, any drop in consistency or change in performance we observe in this zone is guaranteed to be a pure effect of our system design choices . This clean starting point allows us to build a solid foundation before expanding further.
Progressive Expansion: Adapting the Perimeter The Discrete Verifiable Sub-Space is our starting point, not our final destination. The task attributes act as dials: we can relax them one by one to expand our measurement perimeter in a controlled sequence.
Instead of jumping directly to complex, noisy tasks, we expand the scope of our registry step-by-step:
First, we isolate system design effects in a zero-noise environment (the baseline) to measure prompts and configurations cleanly.Next, we introduce stochastically judged tasks , allowing us to characterize how stochastic evaluators (like LLM-as-judge) affect our reliability metrics.Then, we integrate external evidence dependencies , bringing retrieval-augmented workflows (RAG) into the qualified perimeter.Finally, we expand to executable outputs and trajectories , mapping the design space of autonomous agents and tool-calling systems.By using task attributes to adapt our perimeter progressively, we ensure that we only introduce new sources of task noise once we have fully characterized system design effects in cleaner environments.
Expanding the Perimeter: The Three Regimes of Scorability Beyond this baseline, tasks are grouped into three regimes of scorability based on how much noise they introduce into the measurement:
Directly Scorable: Closed response space, single correct answer, and comparison between runs requires no text normalization (for example, multiple-choice or binary verdicts). Variance here is 100% system design signal.Scorable after Normalization: The response is unique, but verification requires surface-level decisions (resolving casing, punctuation, or synonyms). This introduces minor measurement bias that must be documented.Non-Trivial Scorability: Complex outputs (long-form text, trees, code) where defining whether two runs produced the "same" answer is itself an unsolved metrological problem. Measuring system variance on these tasks adds noise on top of uncertainty.The Core Properties in Rhondda The Two Pillars of Rhondda Rhondda evaluates AI systems across two fundamental, non-substitutable core properties: Task performance (does it succeed?) and reliability (how does it behave?). A high performance score does not excuse poor reliability, and vice versa.
Why not just call performance " Accuracy "?
The AI industry often uses "accuracy" as a catch-all word for success. Rhondda refuses to do this for two reasons: • Accuracy is just one metric: You can measure the "accuracy" of a math solver, but you can't measure the accuracy of a creative essay (you'd use a metric like an LLM-as-judge ). Task performance is the broad category; accuracy is just one specific tool inside it. • We don't neutralize it: Some researchers mathematically filter out accuracy to isolate "pure" reliability. Rhondda does the opposite. We treat performance as a fully legitimate outcome that we want to measure head-on, not a mathematical variable to be neutralized.
The Structural Asymmetry These two core properties behave entirely differently as we break them down.
Reliability admits a canonical decomposition into four universal dimensions (adapted from Rabanser et al.): Consistency , Robustness , Predictability , and Safety . Task performance does not. What "accomplishing the task" means depends entirely on the task itself. Thus, performance is broken down into local, task-specific criteria (like factuality, coverage, or format).
Because of this, there is a unidirectional dependency . To measure if a system is robust (maintaining acceptable performance) or predictable (knowing when it will fail), you first have to define what "performance" and "failure" mean for that specific task. You must define performance before you can measure reliability. (Note: A strict boundary remains. Just because an execution constraint helps task success does not make it a "safety" metric, and conversely, a true safety violation must never be diluted as a simple performance failure).
Beyond Autonomous Agents Rabanser originally built these four reliability dimensions for autonomous AI agents. Rhondda proves they apply to all LLM systems. Whether you are evaluating a complex agentic loop or a single, non-agentic LLM call, the system still has a "behavior" that can be evaluated for consistency, robustness, and safety—even if the specific metrics used to score them must be adapted to the lack of sequential trajectories.
The Four-Level Hierarchy To map this theory precisely, Rhondda enforces the following structural hierarchy, visually capturing the asymmetry explained above:
Level For task performance For reliability Core Property task performance reliability Intermediate Level performance criterion , defined by the task (factuality, coverage, format, writing quality...)canonical dimension : consistency, robustness, predictability, safetyMetric accuracy, BLEU, ROUGE, human evaluation, LLM-as-judge, etc. Cout , Renv , Pcal , Scomp , etc. Measure dated application of a performance metric to a configuration dated application of a reliability metric to a configuration
From Theory to Output: Metric, Measure, and Disqualifications Rhondda maintains absolute strictness in how it names the output of an evaluation. An abstract metric is not a measure until it is applied:
Metric Measure abstract object concrete object scoring procedure (formula, rubric, grid, protocol) result of an application exists outside of any execution dated, situated, traceable reusable unique to a case "outcome consistency is a metric" "the measure of outcome consistency on configuration X for task T is 0.64 ± 0.03"
Because we are strict, we ruthlessly disqualify bad data:
A qualitative assertion ("the model seemed smarter" or "the tone felt more natural") is never a measure. A quantified number without a characterized margin of uncertainty is not a qualified measure. If a metric relies on an evaluator (like an LLM-as-judge) that is not fully documented as an experimental instrument, the resulting measure is disqualified because its conditions of comparability are broken. The Final Outputs: Metadata and Effects Separately from the four-level hierarchy above, Rhondda tracks the final outputs of an experiment:
Category Nature Examples Experimental Metadata observed execution variable, mandatory whenever observable, not assimilable to a core property latency, duration, tokens, API calls , inference cost, technical failures Effect derived object: difference (or ratio, or gap) between two comparable measures, attributed to an isolated design factor . Not a level in the hierarchy—it is what the juxtaposition of measures in a controlled comparison produces. Main output of the program. "+0.27 between configuration A and configuration B on the consistency dimension, attributed to the choice of pattern X"
Metadata Usage: Variables like cost and latency are mandatory. They allow us to interpret real-world value. Atlas exposes these costs diagnostically, and Compass arbitrates them based on specific use cases.Main vs. Interaction Effects : A main effect is attributed to a single design factor. When two factors are co-evaluated in a factorial design , the interaction effect is the gap between the observed joint effect and the expected composition of the main effects. (A zero interaction effect is a positive result, proving additivity ). Generalization (Second-Order Characterizations): If you test an effect on a different model, you are measuring a model interaction or a population distribution. If you test it on a different task, you are testing a generalization hypothesis. Neither of these creates a new "core property." They simply test the effect's domain of validity.Special Rules for Reliability
The Safety Rule : Safety is strictly non-compensable. It cannot be mathematically averaged with consistency or robustness to produce a generic "reliability score." A system that is 99% safe but causes catastrophic harm in 1% of cases is not safe. Safety must always be tracked independently. Reliability vs. Validity: A widespread trap is treating "alignment with human judgment" as a dimension of reliability. Rhondda firmly rejects this. Human agreement measures validity (correctness against an external oracle) or serves as a performance metric. Reliability characterizes the system's internal behavior. You must measure a system's reliability first, entirely independent of an oracle, before verifying its validity.Summary FAQ: The Rhondda Vocabulary If you want to speak the language of Rhondda, follow these absolute rules:
Question Rhondda's Answer Is reliability a metric? No. It is a core property. Is consistency a metric? No. It is a dimension of reliability. Is Cout a metric? Yes. A metric of the consistency dimension. Is "Cout =0.64 on config X" a metric? No. It is a measure (an application instance of the metric). Is an observed difference a measure? No. It is an effect, calculated from comparable measures. Is task performance part of reliability? No. It is another core property, at the same rank. Does task performance have dimensions? No. It has performance criteria, defined locally by the task. Is there a canonical list of performance criteria? No. Criteria are task-specific; a canonical list is a category error. Is cost one of Rhondda's core properties? No. It is experimental metadata, mandatory but outside the hierarchy. Is an interaction effect a third core property? No. It is a derived effect from a factorial design. Is variance across models or tasks a core property? No. It is a second-order characterization (its domain of validity). Is safety aggregated with other dimensions? No. It is non-compensable and treated separately. Is human agreement a reliability dimension? No. Reliability characterizes behavior. Human agreement falls under validity, or is a metric for task performance. Is a qualitative assertion a measure? No. A measure is a quantified property with a protocol and uncertainty. Is a quantified number without uncertainty a measure? No. Without characterized uncertainty, it is not a qualified measure.
Why it is not trivial "Imagine you are trying to measure the effect of temperature on a RAG pipeline. You change the temperature, but the pipeline also changes its retrieval threshold, and the model itself behaves differently at each temperature. You are no longer measuring one thing. You are measuring three things at once."
The Problem of Interdependence Breaking down AI system design into 272 atomic design factors is necessary to attribute an effect to a cause, but it is not enough. An atomic factor is rarely an independent one.
In a real LLM-based system, design factors are deeply interdependent. Isolating the effect of a single choice requires controlling an environment that constantly resists invariance. Any rigorous measurement must confront four compounding sources of interdependence, which Rhondda is specifically built to tame:
Structural Interdependence: Changing one design factor often mechanically activates others (e.g., enforcing a specific JSON output format inherently alters the token budget consumed). Rhondda solves this through a strict isolation protocol. When physical isolation is impossible, the registry forces the explicit declaration of those tied variables (treating them as confounding variables ). This strictly bounds the limits of what the measurement actually claims. Model Interdependence: An effect observed on one model might be neutralized or reversed on another. Instead of hoping for universal laws, Rhondda treats the model itself as an axis of interaction. We stratify measurements (via stratified sampling ) across a population of models to map the distribution of the effect. This explicitly proves whether a capability is model-agnostic or tied to a specific lineage. Task Interdependence: A constraint that improves task performance on a multi-step reasoning problem might throttle it on an open-ended creative task. Rhondda anchors every measurement to a characterized task family. The task is never a neutral backdrop; it is a constitutive condition of comparability. Residual Stochasticity: Even with the design factor, model, and task strictly locked down, two identical runs will yield different outputs. This variance comes not only from the LLM itself (sampling seeds, silent API drift over time), but from the surrounding architecture (e.g., retrieval order, cache eviction, concurrent scheduling, varying hardware execution). Rhondda turns this noise into data. We repeat the execution (using Monte Carlo-style repetitions ) to isolate the true effect, and we use that exact variance to measure the Consistency dimension of reliability (providing a formal test-retest reliability ). These four sources do not add up; they multiply. Addressing them simultaneously by building strict configurations, stratifying models, mapping tasks, and repeating runs creates a multiplicative cost. This is exactly why rigorous measurement is structurally expensive, and why so few do it. Rhondda exists to absorb this combinatorial burden upfront.
But absorbing the cost is only the first step. To turn raw data into a coherent coordinate system, Rhondda deploys two structural breakthroughs:
From Disjointed Scores to Composability If we were to test every design factor independently, we would naturally test it in the environment where its effect is most visible. But doing so yields a disjointed collection of scores that share no common ground and cannot be compared.
To solve this, Rhondda does not collect isolated effects. Instead, it enforces a Composition Perimeter. For a given cycle of measurements, we freeze the underlying conditions: the task family, the model, and the structural regime are strictly locked. Because the context is shared, every measurement within this perimeter becomes composable.
For example, instead of vaguely asserting that temperature and output formatting both impact performance, this shared context allows us to prove their relative weights: that at a fixed model and task, enforcing a JSON format costs exactly twice as much as raising the temperature.
From Isolation to Interpolation But enforcing a shared context only bounds the problem; it does not solve the fundamental issue of non- additivity . When two interdependent factors are used together, their combined effect is rarely the simple sum of their isolated effects. The penalty of a JSON format might double if the temperature is high.
Rhondda does not treat this non-additivity as a nuisance to be ignored. Instead, we measure the gap.
We evaluate each factor alone, then the joint configuration (deploying a factorial design ). The gap between the observed joint effect and the simple sum of their individual effects is the interaction effect .
Interaction effects are treated as fully legitimate derived effects in the registry. When the interaction effect is zero, we have proven additivity. When it is non-zero, we have quantified their synergy or antagonism.
This is the missing link that tames interdependence. By providing the exact gradient between data points, Rhondda's interaction effects allow us to reliably interpolate the outcome of a compound architecture, including its expected reliability and quantified uncertainty, rather than blindly hoping isolated properties will stack cleanly.
The Final Output: Actionability Scientific rigor alone produces results that are true, but limited to their lab conditions. A designer's intuition produces systems that are applicable, but fragile.
It is strictly the combination of Composability (knowing how isolated levers stack) and Interpolation (safely applying known effects within a characterized family ) that bridges this gap. Together, they transform theoretical measurements into actionable engineering: allowing a designer to predictably build a complex system, without having to empirically test every single combination themselves.
Where Rhondda fits in the literature The academic landscape possesses powerful benchmarks, exemplary infrastructures, theoretical frameworks, and convincing factor-isolated studies. Rhondda does not invalidate these achievements. It re-indexes them around a specific missing unit of accumulation: the qualified effect of an atomic system-design choice, measured at constant conditions, with its inter-run variance , quantified uncertainty, white-box noise floor , and published in a composable registry.
To understand Rhondda's exact position, it must be mapped against the specific technical contributions of the existing literature.
Infrastructure and Standardized Evaluation: HELM & HAL HELM provides the foundational posture: public registry, standardized schema, and transparency. However, HELM characterizes models, not design effects. It fixes the system design to compare models, whereas Rhondda fixes the model to measure the effect of design choices. When HELM observes a massive effect from a design factor , output format, prompt wording) whose effect can be isolated and attributed. The v0.1 grid contains 272 design factors. Kapoor's HAL is the reference for agentic infrastructure (harness, orchestration , cost tracking). While its base leaderboard focuses on task success, its Reliability Dashboard explicitly measures reliability dimensions. However, HAL evaluates entire multi-factorial scaffolds rather than isolating individual atomic design factors , and it typically relies on min-max ranges rather than formal confidence intervals across runs. HAL is the harness and leaderboard layer; Rhondda is the isolated design-effect registry layer. Reliability Frameworks and Measurement Theory: Rabanser, Chehbouni & Khanal Rabanser et al. provide the operational vocabulary for reliability, correctly identifying that accuracy is not enough. However, their empirical work relies on black-box , and replaying generations deterministically, they tighten uncertainty from a statistical interval into a verified, reproducible fact—moving from correlation to causation without relying on the asymptotic assumptions that bootstrap CIs require. LLM-as-judge dependencies. Rabanser provides the y-axis; Rhondda provides the x-axis and the empirical warrant. Chehbouni et al. offer the theoretical diagnosis and measurement theory framework. Rhondda translates this normative call into shared instrumentation. Khanal et al. provide strong directional validation, using bootstrap CIs . However, the scope remains on long-horizon agents with non-orthogonal metrics. Rhondda tightens the uncertainty and moves to single-call , white-box observability. Surveys and Field Maps: O'Brien & Gu et al. O'Brien uses the closest lexical neighbor to Rhondda (factors, levels, conditions, system-under-test). However, O'Brien ranges users, tasks, documents, systems, and outputs on the same "factor" plane. Rhondda strictly separates the input (design factors), output (dimensions), and conditions (task, model, metadata).Gu et al. position reliability as a central thread but fuse it with validity (alignment with human judgment). Crucially, their own experiments explicitly suppress variance (setting T=0 "to reduce randomness" with a majority@5 vote) rather than measuring it, leaving the actual noise of the system undocumented.Factor Isolation & Bias Quantification: MAFBench, ReliabilityBench, Zhang, Shi, Ye & Do Xuan Long While factor isolation exists in the literature, it often lacks atomic granularity, cross-run confidence intervals (CIs), or targets the wrong locus:
MAFBench & ReliabilityBench isolate choices at the framework or bundle level (e.g., planning, topology, ReAct vs. Reflexion), producing point-estimates without inter-run variance.Zhang et al. (Format Bias) provide the cleanest isolation via minimal pairs, but measure the validity of the instrument (proving the LLM-judge is confounded by format), not the reliability of the system under test, and rely on point-estimates without CIs.Ye et al. (CALM) & Shi (Position Bias) measure the first "proto-noise-floors" of LLM judges (Consistency Rate and Repetition Consistency). However, they use these metrics in black-box settings without CIs, primarily as a gate to justify single-run evaluations, rather than as a qualification threshold to attribute an effect. Do Xuan Long (Output Format Bias) shares Rhondda's exact unit of analysis: the effect of an atomic design factor (output format) at a constant model in a single-call setup. However, it relies on single runs under stochastic decoding , with no confidence intervals across runs and no white-box exploitation. Its variance is measured between formats, not across runs, leaving the consistency dimension unaddressed. Technique Re-evaluation: Self-Consistency Wang et al. (Self-Consistency ) established that majority voting improves average accuracy. Rhondda re-evaluates this technique by measuring its effect on the consistency dimension itself, using inter-run variance as the primary output. A preliminary pilot on GSM8K proves why this shift matters: Rhondda successfully isolated a subset of items (9%) where the model achieved perfect consistency but zero accuracy—a systematic, reproducible failure that accuracy-centric evaluations mistake for random noise (detailed in the Rhondda DOES work section). In summary: The distinctiveness of Rhondda is the conjunction. The field has isolated variables at the framework level, measured proto-noise-floors on judges, and mapped reliability theoretically. Rhondda brings atomic isolation, constant models, quantified uncertainty, white-box observability, and inter-run variance together into a single, composable registry.
And finally... the System Design Grid itself To be clear: no reviewer is expected to read a 272-factor taxonomy (!!), though I've tried to make it as easy to explore as possible given the format. Making this raw data truly navigable and diagnostic for practitioners is the exact UX challenge that the Atlas interface is meant to solve.
Functionally, this grid divides the architecture of any LLM-based system into three levels of abstraction:
Level Count Definition Families 6 The high-level functions of the system (e.g., Context Construction, Orchestration ). Crucially, Rhondda does not have an "Agent" family . An agent is simply a specific co-activation of these exact same foundational families . Zones 38 Groupings of choices by homogeneous structural function. Design Factors 272 The atomic, isolated variables that we actually measure.
I have included the grid here simply out of curiosity, so you can see what it looks like in practice. Feel free to search for a keyword or unfold a family to get a sense of the granularity.
Family A — LLM Input & Context Construction
Family B — Interaction with Resources & Environments External to the Current LLM
Family C — Current LLM Inference
Family D — Output Processing, Validation & Selection
Family E — Orchestration, Control Flow & Execution Policies
Family F — Continuity, Memory & Persistence
What is Atlas? Imagine you are building an LLM-based application and have to select its default parameters. You know that for your use case, the temperature shouldn't be 0, but you do not know if these settings will degrade your model's accuracy, how much variance they will introduce, if your specific AI model behaves differently from the framework's baseline, or what other design factors are impacted.
To find out, you would have to spend money and time running repetitive, multi-model testing sweeps just to establish a baseline noise floor (the baseline fluctuation of the model).
Instead, you open Atlas. It tells you where to start and gives you directions.
Atlas is a navigable and readable version of Rhondda's registry.
To bridge the gap between academic research and engineering practice, Atlas is structured around two distinct, interconnected views pointing to the exact same dataset: the Coordinates View (the structural grid) and the Designers View (the practitioner gallery).
The Coordinates View The Coordinates View is the structural grid of Rhondda's taxonomy. It maps the six families and Rhondda's empiric datas into a navigable, action-oriented decision matrix. The core objective is simple: to make this extensive registry immediately readable and easily navigable, saving designers from guessing parameters or re-running expensive testing sweeps from scratch.
How it will look like Instead of static listings, Atlas projects the entire taxonomy as an interactive 2D matrix (similar to the MITRE ATT&CK ® matrix). Here is a concept mockup of how I imagine it will look (this is a conceptual rendering, not the live application):
For example, one can imagine that selecting a zone would also automatically highlight all its dependent zones in green on the matrix grid.
The Interaction Layer A designer thinks in interactions as much as in isolated effects (e.g., 'which aggregation rule for which $k$?'). Atlas overlays a dynamic interaction layer representing the couplings graph from the registry (~959 links source_df → targets with polarity and evidence). To prevent visual noise ('hairballs'), this layer is never shown all at once: • Voisinage (Local Network): Hovering over a choice highlights its immediate local network (3 to 5 links) and opens a sidebar panel listing its interactions. • Active Surfaces (Cumulativity): Clicking a measured interaction link opens the configuration landscape. An interaction surface is built by stacking slices of isolated measurements (e.g., measuring $k$ under vote, then under best-of, then under consensus). Stacking these slices constructs the surface. • Roadmap: Predicted couplings that are not yet empirically measured are displayed as dotted lines, rendering the program's roadmap explicit.
Family Definition (Summary) Zones Factors A: Input & Context What the model sees at input and its structural assembly. 11 91 B: External Interaction Access to resources and environments outside the current inference call. 6 59 C: Inference The execution of generating a completion from a prepared input. 5 22 D: Output Processing Transforming, validating, filtering, or selecting output candidates. 5 26 E: Orchestration When, how, and in what order calls occur, loop, retry, or repeat. 6 36 F: Memory & Persistence What persists across steps, conversation history, and indices. 5 38
The Designers View The Designers View is the practitioner's gallery. A system designer usually does not start from a taxonomy grid; they start from a concrete question:
"Is it worth using a technique like self-consistency ? What does it imply for my system?"
To answer this, Atlas represents every major technique or configuration as a Design Choice . Designers can search for specific choices, explore their isolated effects, and critically, combine multiple choices together to reveal their interdependencies. If the Rhondda registry contains empirical data on these combined regimes, Atlas will map the resulting interaction surfaces.
Dynamic Organization via Facets Instead of rigid folders, the sidebar organizes the gallery using dynamic facets over a single list. The gallery can be grouped by: • Family: A strict partition where each choice resides under the family of the lever it varies (e.g., Inference, Input & Context). • Regime: Grouped by execution complexity ( Single-call , RAG , Agent). • Magnitude Affected: A choice appears under every performance or reliability dimension it shifts.
The Active Workspace: Varying and Co-activating Instead of static documentation, every choice opens an Active Workspace representing the configuration as a node graph. Because a design choice is rarely perfectly isolated in practice, this workspace spatializes the dependencies: • The Central Node (Varying): The specific design factor that this choice manipulates (e.g., Candidate count for Self-consistency). • The Connected Nodes (Co-activates): The other design factors that are mechanically activated or must be explicitly held constant alongside it (e.g., the voting aggregation rule, or the stochastic sampling parameters).
Empirical Evidence & Model Specificity Selecting a choice also reveals its dedicated empirical panel containing the load-bearing data from the Rhondda registry. This provides the exact metrics needed for engineering: • Empirical Test Curves: Direct visualization of the trade-offs, such as Accuracy & Cost vs. Candidate Count ($k$). • Model Specificity: A qualitative synthesis of where the effect holds (e.g., identifying when gains plateau or which model lineages show the steepest curves). • Taxonomy Registry: The formal metadata (Attribution Ceiling, Atomicity, Precondition Load) ensuring the boundaries of the measurement are explicit.
The Trajectory of Filling Atlas is populated in the order of isolability of the program (from the most isolable to the most co-activated): • Single-call: Inference (C) + one point in Input (A.10) and, later, in Output. • LLM-as-judge : External evaluation (D downstream) + input construction (A prompt), involving a separate judge model call to score or compare outputs. • RAG: External Interaction (B.1 retrieval), Memory (F.2 build-time: chunking/embedding/index), Input (A.7 context), Output (D downstream). • Agent: Orchestration (E) last , the most co-activated and least isolable regime.
What Atlas does NOT do Atlas provides the public decision grid (cross-model distributions, variance, noise floors, gain/cost surfaces). It does not prescribe which configuration is optimal or 'plant the flag'.
The application to a specific custom stack (such as calculating net gain under proprietary constraints, prioritizing mitigations, or setting specific action plans) remains the responsibility of the system designer. The boundary is strict between the generic empirical baseline (what Atlas provides) and custom application (what you build):
The Question Atlas Answers How it Works (Empirical Baseline) What Atlas DOES NOT Do (Strict Limits) What do we know about this regime? Filtering choices via the Regime facet (e.g., RAG vs. Agentic) immediately reveals which design factors are active and relevant to a specific architecture. Does not prescribe which architecture you should choose or recommend a specific regime. What do we know about this design choice? Isolates the choice's effect on task performance (accuracy ) and reliability dimensions (consistency , robustness , predictability , safety ). Does not predict how a choice will behave under unknown tasks or unmeasured context. What are the interdependencies? Projects the couplings graph on the Interaction Layer, showing how changing one design factor modulates or conflicts with the effects of another. Does not automate the resolution of engineering conflicts or design trade-offs. In what proportion does the effect manifest? Exposes the magnitude of the effect and its distribution across a panel of at least 16 reference models (by lineage, scale, and reasoning capabilities). Does not ingest custom user models or compute custom client baselines. How does the performance curve behave across configuration ranges? Plots multi-dimensional performance surfaces (e.g., candidate count k vs. accuracy and token cost) to show where gains plateau and where variance dominates. Does not label configuration intervals as "optimal" or "plausible defaults."
What you can derive from Atlas's curves: Because Atlas exposes the raw curves and noise floors rather than labeling them, you are empowered to: • Identify Pareto-efficient trade-off zones : Locate the exact "sweet spots" where performance returns diminish and token costs or latency begin to explode. • Challenge framework defaults: Validate and de-bias arbitrary default settings frozen by popular AI libraries and orchestrators. • Establish baseline noise floors: Determine the minimum number of evaluation runs required to distinguish a real system improvement from statistical noise in your own tests.
NOTE: Atlas has no user persistence, no production monitoring, and no pipeline integrations. It remains a completely open, exploratory metrological map answering these design questions for the community.
Why should Rhondda exist Take a team trying to build a customer-support AI.
They have to decide whether the model calls the CRM directly or deterministic code routes the call, whether the model sees every past ticket or just the last three, whether to use RAG or agentic search or both, whether to prompt for caution or for resolution on first contact. Each of these choices reshapes the architecture and shifts the system's performance and reliability, and at this stage, before the whole thing has been tested, each one is a bet placed on intuition. There is simply no shared evidence to consult.
I've spent months asking a simple question: when a team builds an LLM-based AI system for production, what do they actually base their design decisions on?
The answer has turned out to be uncomfortable.
Something is wrong with AI systems design Every day, across the industry, teams make choices that determine, at first order, whether a system performs and holds under real-world variation. What goes to deterministic code and what goes to the model. Which architectural pattern to adopt: RAG , tool-calling agents , deterministic workflows enriched by LLM, multi-agent setups, or some hybrid. Which model to use. How to write the system prompt , design the output schema , order the tools, set the inference parameters, wire the guardrails .
None of these are marginal, and practitioners debate them fiercely. When Anthropic published its guide to building effective agents in late 2024, the question it chose to anchor the entire piece was "should I use agents or workflows?" The answer spans a full architectural philosophy — the stakes are real, the evidence thin. "The range of possible system designs for a given task is vast." Zaharia et al. wrote that sentence as a diagnosis in 2024. Vast, and entirely unmeasured. And yet, for all this intensity, the field cannot tell you with controlled evidence what effect any of these choices produce — not on a characterized task family, not at fixed model, not with documented conditions. Gu et al. , whose 2026 survey of LLM-as-judge research is the most comprehensive to date, put the point plainly: it is often challenging to isolate the bias of interest from other confounding factors. They conclude that there is a pressing need for a unified benchmark that systematically and comprehensively quantifies these biases within a single framework. The field's own cartographers are telling you the map doesn't exist. We are operating in a pre-empirical regime This is not a data problem. Teams produce enormous amounts of local empiry: internal tests, ad hoc evals, production logs, variant comparisons. The problem is that none of it is comparable, because every team measures in its own schema, against its own tasks, under its own conditions, and local data stays local. Without a shared substrate to consult, here is what teams arbitrate on instead.
Personal experience from prior projects, except those projects ran under different, usually undocumented conditions, so the experience does not transfer.Expert opinion , calibrated on a handful of incomparable projects under implicit conditions that cannot be audited.Methodological conformism and fashion cycles: yesterday RAG, then agents, then multi-agent, tomorrow something else — turning faster than evidence can form, with generalization neither measured nor proven.Anecdotal war stories from blog posts, conference talks, and podcasts, structurally biased toward successes and toward configurations that sound good on stage, conditions left implicit.Vendor recommendations , which carry a structural conflict of interest on the very object being measured.Framework defaults , which freeze the editor's arbitrary choices into values that stop being visible as choices at all.Fragmentary academic literature , rigorous in isolation but with conditions that don't overlap across studies. Angermeir et al. put a number on this at ICSE 2026: they examined 85 LLM-centric studies from recent top-tier software engineering venues, found 18 with research artefacts, 5 executable enough to attempt reproduction, and achieved zero complete reproductions. Their phrase for what the field produces: time-bound artefacts rather than building blocks of cumulative scientific knowledge.The intersection of these seven sources is not an empirical substrate. They do not speak to each other, so they cannot be weighed against each other. What emerges is a marketplace of opinions where the loudest voice, the best-funded vendor, or the most prestigious title wins, not because anyone is wrong, but because there is no shared ground for a different kind of argument.
What it feels like to be an AI system designer The abstraction "the field lacks a substrate" is too bloodless to convey what it actually costs. So let me describe what designers do in the face of this uncertainty — three concrete pains, and every practitioner knows them.
1. The unbounded space. The question "can this perform better?" has no answer without sweeping combinations, and the space is, in practice, infinite. But not because there are many parameters. It is infinite because the parameters are interdependent: the effect of one choice depends on the values taken by all the others, which means you cannot optimize them independently and combine the local optima.
The designer tests a handful of variants around the current setup, stops at "good enough," and has found a local optimum without being able to recognize it as local. They do not know what they haven't tested, and even what they have tested doesn't tell them why it worked.
2. Holistic measurement, not attributed. The verdict bears on the entire configuration, not on the effect of any single choice. The designer knows the whole thing holds or fails, and to what degree, but not which choice carries the result, which means they cannot identify the levers, not to optimize further or to fix the system when it breaks in production.
Isolating the effect of a single choice would require varying it with all else held equal, and this almost never happens. The reasons compound.
Combinatorial cost: you multiply the number of factor levels by the repetitions needed to beat the noise floor , by the number of factors, by the token and time cost of each run. The product explodes fast. No ready-made tooling: eval frameworks measure whole configurations, not isolated factor variation.Product pressure: shipping a working config beats understanding why it works.3. The noise floor. Even the holistic verdict is only as good as the metrology that produces it, and this is where the abstraction hits the ground.
Consider a team choosing between two rerankers . Reranker A, the current one, scores 0.71 on 200 questions. Reranker B, the new one, scores 0.74. The tempting conclusion writes itself: B is better by 3 points, switch to B.
What is missing from this conclusion is the variability of the measurement itself. The system is not deterministic. Re-run A without changing a single parameter and you might get 0.71, then 0.68, then 0.75 — an oscillation of ±0.03 with nothing touching the system. That is the noise floor, and the "gain" from B sits squarely inside it — noise was mistaken for a system effect.
And the worst part is you cannot see it. The conclusion has a score, two decimal places, 200 questions. It looks rigorous but it is not.
You would think the research community, having named reliability as a central concern, would at least measure this noise floor. In most cases, it does not. Gu et al., the survey that places reliability at the center of the LLM-as-judge field, sets temperature = 0 in its own experiments explicitly "to reduce the impact of randomness," then applies majority voting over 5 runs on top, suppressing the variance at two stages without measuring it once. The field's own best map of the problem erases the quantity that defines it.
To be rigorous you would need to repeat each configuration, measure the dispersion, and only conclude "B > A" when the gap clearly exceeds the noise. That explodes an attribution cost that was already prohibitive. It is rarely paid, and nothing enforces it.
Taken together, these three pains define the trap. The configuration space is unbounded, so the designer does not know where to look. Even if they did, holistic measurement does not tell them which lever moves the needle. And when they do pay for rigor, the metrological cost to make measurements actually mean something explodes the budget.
Even if they pay all of that, what they establish dies with their project. The data does not accumulate, does not grade itself, does not compose. Three verbs, three missing functions. No amount of local effort fills any of them.
What the field cannot know Here is the consequence, stated plainly.
The field cannot know which production design decisions were suboptimal. It cannot know which reliability failures could have been anticipated with even modest controlled comparisons. It cannot know which deployed architectures would have been eliminated by the evidence, had that evidence existed.
Deeper still: it cannot even establish whether these questions are worth asking, or at what scale. The ignorance is not just about answers. It is about the visibility of the questions themselves.
What would it take to change this? Not an oracle. You do not need the substrate to tell you "choose X." You need it to tell you, in conditions close to yours, that this lever has this directional effect — and to know the sign, the plausible magnitude, the confounders to watch for. An effect that is dated, conditioned, graded by confidence, and revision-tracked. Not a prescription. A qualified starting point.
That is the gap, and it is structural. Not because no one is trying, but because no one carries the function of filling it.
What we have is not enough The landscape that currently produces empirical material on LLM-based AI systems is not empty. Several families of actors operate within it, each with a distinct function and sometimes excellent results within that function. None carries the function Rhondda assumes, and it is this functional asymmetry that must be made legible.
What follows is not a criticism of these actors. It is an examination of their perimeter. The question is not whether they do what they do well. The question is whether the sum of what they do reconstitutes an empirical substrate on the effects of system-design choices at controlled variables. The answer is no, and this section explains why, family by family.
1. Model Evaluation Platforms
Platforms like HELM , the Open LLM Leaderboard, and Artificial Analysis evaluate foundation models in isolation on standardized scenarios. HELM proved a public, standardized registry can structure a field — a precedent Rhondda builds on explicitly. But these platforms characterize models . Rhondda measures effects of system-design choices in controlled comparisons. A HELM measurement says nothing, by construction, about the effect of changing an architectural pattern on the same task at constant model. No sum of HELM measurements reconstitutes a Rhondda comparison. This is not a shortcoming in HELM. It is a perimeter boundary — one Rhondda does not cross in reverse.
2. Agent Evaluation Platforms
The Holistic Agent Leaderboard ( HAL ) profiles composed agents on benchmarks like GAIA, τ-bench, and SWE-bench. Its Reliability Dashboard, built from Rabanser et al. 2026 , measures agents across four reliability dimensions — consistency , robustness , predictability , safety — with cost-aware metadata. Rhondda adopts this vocabulary and discipline explicitly. What separates Rhondda from HAL is not methodological quality but unit of analysis. HAL ranks agents ; Rhondda registers the effect of a single design choice isolated in a controlled comparison. The two are complementary — a HAL profile can inform a Rhondda comparison — but they are not substitutes. An agent leaderboard cannot stand in for a registry of design effects.
3. The Compound AI Systems Community
In February 2024, Zaharia et al. (Berkeley and Databricks) published The Shift from Models to Compound AI Systems , naming a community around a diagnosis Rhondda shares: state-of-the-art AI results come from compound systems, not models alone, and the range of possible system designs for a given task is vast. Rhondda inherits this diagnosis. What it rejects is the response. This community answers the design-space challenge with optimization tools — DSPy , prompt optimizers, model routers, continuous eval methods — that tune a particular system toward a target metric. DSPy does not tell you whether a deterministic LLM-enriched workflow is dominated by an autonomous agent on a given task family, at a given model, under a given cost regime. Tooling the design and measuring the effects of design choices are two responses to the same diagnosis, but they are orthogonal. Rhondda measures what the compound AI systems community tools.
4. Academic Papers
The academic literature continuously produces isolated comparisons — RAG versus fine-tuning , chain-of-thought versus direct answer, single-agent versus multi-agent — that are often methodologically excellent taken individually. Rhondda does not dismiss this work. A rigorous paper comparing two architectural patterns at fixed model, on a characterized task family, can, under the registry's qualification standard, become a registry entry. Rhondda does not substitute for distributed academic production; it provides the framework within which that production can accumulate.
What Rhondda rejects is the idea that this literature spontaneously aggregates into an exploitable substrate. No common schema. No shared versioning. No homogeneous qualification standard. No programmatic memory. Every paper remains an island — not through any defect of individual researchers, but because academic incentives do not reward methodological cumulativity and no one carries the infrastructure that would make it possible. Each paper is a time-bound artefact rather than a building block of cumulative scientific knowledge.
5. Industry Experience
The abundant material of industry returns — editor blogs (Anthropic, OpenAI, Cursor, Databricks), sector conferences, technical podcasts, public post-mortems, vendor best-practice guides — transmits practical experience, often valuable: choices that worked, choices that failed, deployment contexts, constraints encountered. Rhondda retains this as monitoring material. Some returns signal methodologically relevant questions to investigate, identify zones where the empirical need is urgent, or orient the prioritization of the program's measurement campaigns.
What Rhondda rejects is their epistemic status. None of these returns satisfies the registry's qualification standard: no controlled protocol, no explicit comparability, publication conditions that structurally bias visibility toward successes and valorizable configurations, and sometimes direct commercial dependency — editor blogs also serve promotional functions. This limitation is a criticism of no one. Production teams have neither the mandate, the incentive, nor the format to produce publishable controlled comparisons. Industry experience feeds Rhondda's monitoring. It does not feed its registry.
6. Instrumentation and Local Eval Platforms
A heterogeneous family of tools has formed around instrumenting LLM-based AI systems in production: tracing (Langfuse, LangSmith, Helicone), eval management (Braintrust, Arize, Patronus, Galileo), prompt and version management, LLM observability and monitoring, internal experimentation platforms. Their function is to equip teams to produce their own local empiry — comparing configuration variants, tracking production metrics, detecting regressions, instrumenting call chains. These platforms are precisely what makes teams capable, in principle, of producing documented comparisons on their own systems, and a portion of external contributions to the registry will likely pass through them.
What Rhondda rejects is the idea that the aggregation of instrumented local empiry itself constitutes a field-level substrate. Schemas remain proprietary or non-standardized. Protocols are not shared. Comparability conditions are documented only within the organization that uses them. Qualification is an internal decision rather than a public act. These platforms instrument local empiry. They do not transform it into public, qualified, cumulative knowledge. That distinction — between an instrumentation tool and a registry function — is exactly what this section makes legible.
7. Standards, Regulators, and Public Safety Institutes
A final family operates at a distinct institutional level: national and international agencies (NIST , the UK and US AI Safety Institutes, ISO /IEC ), sectoral regulators (health, finance, transport), and industry consortia publishing governance frameworks. Their function is normative and prudential: producing risk-management frameworks, failure taxonomies, transparency requirements, audit and red-teaming protocols. When these actors define reliability dimensions to measure or risk categories to investigate, they signal zones where the empirical need is institutionally recognized — and the substrate Rhondda carries can, in turn, feed their work. What Rhondda rejects is the idea that the normative function can stand in for the empirical one. Defining a governance framework is not producing controlled comparisons. Requiring that a system be evaluated is not producing the evaluation or qualifying its method. Publishing a risk taxonomy is not measuring the effect of a design choice on that risk. The normative function and the empirical function are compatible and complementary — they are not substitutable. Rhondda can feed this family's work without replacing it; this family can orient Rhondda's priorities without inheriting its mandate.
What is missing is a function, not a quality
The existing landscape is not empty, and Rhondda does not claim to fill a desert. It inherits partial functions from every actor mapped above: HELM shows a public evaluation registry can structure a field; HAL and Rabanser et al. provide the operational vocabulary of agent reliability; the compound AI systems community names the design-choice space; academic literature produces results qualifiable under the registry standard; industry experience signals where the empirical need is most acute; instrumentation platforms equip teams to produce their own local empiry; standards and regulators normatively map the field.
What none of these actors carries is not a methodological quality they lack, but a function that is not theirs: to produce, qualify, version, and maintain a public registry of comparable measurements on the effects of system-design choices — typically at fixed model when the observed choice is not the model itself — on characterized task families. This function is not the sum of theirs. It does not reconstitute itself by adding their contributions together. And it is this function — distinct, dedicated, durable — that Rhondda exists to carry.
That is the gap the next section examines: why it is structural, and why it will not close by itself.
The solution won't emerge by itself The previous section established that the existing landscape does not fill the gap. The harder question is why it won't : why this function does not reconstitute itself spontaneously from the contributions of the actors already in place.
The answer is not that these actors do their work poorly. Each operates rationally within its own incentives and constraints. The gap exists because no actor carries, in its durable mandate, the function of producing, qualifying, versioning, and maintaining a structured empirical substrate on the effects of system-design choices , at controlled variables. This section establishes that fact, actor by actor, and draws the consequence.
The Academic Incentive Regime
Academic production on LLM-based AI systems is abundant and sometimes excellent. It does not spontaneously accumulate into an exploitable substrate. Three structural mechanisms are responsible.
1. Originality is rewarded; cumulativity is not. Journals and conferences value novel contributions, new methods, new results. A study that methodically establishes, under a shared standard, an effect already published is rarely publishable. A replication, however well-conducted, does not build a career. This incentive structure is rational for identifying original contributions, but it penalizes precisely the work a cumulative substrate needs.
2. Faster diffusion is not cumulativity. Preprints, public repositories, dashboards, and living benchmarks have shortened dissemination cycles. But diffusing a result faster does not make it cumulable. Cumulability requires a common schema, qualification criteria, versioning, a memory of revisions, and the capacity to reinterpret measurements when models, tasks, or protocols change. Academic production can accelerate without ever becoming a registry function.
3. The agenda is captured by capability races, not design science. The community's attention is largely absorbed by new models. The system-design choices that govern their exploitation receive less dedicated, systematic attention. This is not the primary mechanism; it follows from the first two but amplifies them.
4. Academics are not rewarded for producing actionable results. A researcher's incentive is to satisfy peer reviewers, not to equip system designers. What counts is whether a finding is defensible under scrutiny, not whether it is characterized well enough for a practitioner to act on it. The effect of a design choice can be reported as a point estimate on a single model, under conditions declared but not verified, without a quantified noise floor , and still clear the publication bar. A designer who tries to build on that result has no way to judge whether it transfers to their system. The researcher has no professional stake in that transfer problem. They already got the publication.
These four mechanisms share a common root: the academic incentive regime rewards defensibility , not actionability . A distinct consequence follows: the gap between publishable and load-bearing . A comparison can be publishable (defensible enough to pass peer review) without being load-bearing, meaning characterized well enough for a system designer to rely on it in a production decision. An effect reported without a quantified noise floor, as a point estimate on one or two models whose generality is unestablished, or under declared rather than verified execution conditions, satisfies the publication standard without reaching the usage standard. The epistemic bar that governs publication (defensibility) is not the one that governs system construction (actionability). This gap is not closed by accumulation: an aggregate of non-load-bearing measurements remains non-load-bearing. It is a reason distinct from, and prior to, the fragmentation addressed by the four mechanisms above, why the sum of academic contributions does not reconstitute an exploitable substrate.
The consequence: academic production generates valuable fragments, some of which can be qualified and integrated into a third-party registry. But the academic community, in its current configuration, does not spontaneously carry the continuous function of qualifying, versioning, and maintaining a cumulative registry.
The Product-Team Regime
Teams deploying LLM-based AI systems into production accumulate enormous practical experience on the concrete effects of system-design choices. This experience does not become substrate. Three structural mechanisms explain why.
1. Publication asymmetry. The conditions under which teams publish (product communication, recruitment signaling, internal valorization) structurally orient what is made visible toward valorizable configurations, at the expense of the informative failures that would characterize what does not work, and under which conditions. Even the configurations that are made visible are rarely described with the methodological precision a controlled comparison requires. A blog post or conference talk is a communication format, not a reproducible protocol. These structural constraints bias what becomes accessible as a corpus.
2. The opportunity cost of methodological rigor. Documenting a controlled comparison to a shared qualification standard demands an investment that does not directly serve product delivery. For a team constrained by an operational calendar, this investment is rationally deferred, often indefinitely. This is not a failure of scientific ethics. It is the effect of a function that is not part of the team's mandate.
3. Confidentiality and competitive advantage. A significant portion of the most useful learning comes from real systems, internal data, client constraints, business metrics, and operational failures that teams have neither the incentive nor, often, the right to publish. Some elements of production experience are structurally non-publishable.
The consequence: the practical knowledge accumulated by production teams remains valuable as monitoring material. But it does not become substrate without a third-party actor qualifying it to a shared standard, and that third-party actor does not exist within the teams' own mandate.
Model Providers and Evaluation Platforms
Model providers and public evaluation platforms produce a significant share of what is currently measurable from the outside on LLM-based AI systems. Their respective positions, which differ from each other, explain why neither can, in its current configuration, constitute the public, durable, methodologically neutral carrier of this function.
Model providers face a structural conflict of interest on the very object to be measured. Measuring the effect of system-design choices on a model's performance and reliability is rarely neutral when you are that model's editor. Model cards, system cards, and technical reports are artefacts of partial transparency: valuable as contextual raw material, but they are not independent controlled comparisons. This does not disqualify their contribution. It disqualifies the idea that they could occupy a public, durable, and neutral position relative to the choices they instrument.
Public evaluation platforms face a functional perimeter limitation. As mapped in What we have is not enough , each platform has its own object (characterizing models, profiling agents) that does not intersect with measuring the effects of system-design choices at controlled variables. HELM characterizes models. HAL profiles agents. No public platform currently carries the mandate to produce and maintain a registry of controlled comparisons between system-design choices. This is not a design defect. It is a perimeter boundary they fixed for themselves, and it does not cover Rhondda's object.
The consequence: providers and platforms can produce measurements useful to the substrate, and Rhondda must articulate with them. But they cannot, in their current position, constitute its public, durable, and methodologically neutral carrier.
Why the Gap Is Structural, Not Cyclical
Four angles make visible that this gap does not close by itself.
1. Mandate. The function of producing, qualifying, versioning, and maintaining a cumulative registry of controlled comparisons on the effects of system-design choices, at controlled variables, is in no mapped actor's mission. This is not a shortcoming on their part; their respective missions are valid within their own frame. It is the definition of a functional lacuna: the function has no designated carrier.
2. Addition. Even under the hypothesis that every actor published perfectly within its frame, the sum of their productions would not yield a Rhondda registry. What would still be missing is the common schema, the homogeneous qualification standard, the shared versioning, the maintenance across time: everything that distinguishes a cumulative substrate from a collection of fragments. The Rhondda function is qualitatively distinct from the sum of its possible parts.
3. Time. A transient delay would close as the field ages. A structural gap does not close through its own inertia. The pace at which new configurations deploy is not accompanied by the establishment of a cumulative qualification function; and the mechanisms described in the three subsections above contain nothing that would spontaneously orient the actors in place toward this function.
4. Maintenance. This is the least visible angle and the most decisive. Producing an initial measurement is an act; maintaining its legibility across time is a function. As models change, benchmarks age, patterns recombine, protocols must be revised, and new reliability dimensions emerge in the literature, the registry survives only through continuous work of qualification, versioning, and re-indexing.
This work is continuous, costly, unspectacular, and weakly rewarded by the ordinary incentives of the field, academic or industrial. No actor mapped in What we have is not enough carries it in their current mandate. If one of them decided to carry it durably, they would no longer be doing so from their present function alone; they would become, de facto, an equivalent carrier of the Rhondda function.
This is why the Rhondda function demands a dedicated carrier: not because the initial production is beyond the reach of the actors in place, but because maintenance, without which a registry quickly ceases to be a substrate, is in no actor's mandate under their current configuration.
The Consequence: A Dedicated Program Is Necessary
No actor carries, in its durable mandate, the function of producing, qualifying, versioning, and maintaining a structured empirical substrate on the effects of system-design choices, at controlled variables. A dedicated carrier is necessary. That carrier must meet at minimum five characteristics:
An explicit mission covering all four registry functions: production, qualification, versioning, and maintenance.A durable public posture , distinct from a time-limited project. A registry that disappears after three years is not a substrate.Independence from model providers and evaluation platform editors. The object to be measured must not coincide with the interest of those who instrument it.A methodological discipline compatible with a shared qualification standard. Without a common threshold, cumulativity collapses into compilation.An explicit articulation with distributed contributions rather than a posture of substitution. No single program can cover the entire design space alone.These five characteristics do not only specify a viable carrier. Under a different reading, they are also the conditions under which such a carrier avoids the risk proper to any methodological infrastructure: producing the appearance of scientificity rather than its substance.
The necessity demonstrated here is functional, not personal. What the field needs is a dedicated carrier, not a particular carrier. Whether that carrier is Rhondda is a question of historical opportunity: who chooses to occupy this function. Rhondda exists because this function demands a dedicated carrier; within the Cubits corpus, it is the form that carrier takes.
People ALREADY WANT (something like) it The same gap runs all the way up, it is structural, and it is scattered across the literature in plain sight, in pieces that do not cite each other. In other words: the people who will use this substrate are already asking for it, they just do not have a name for what they are asking for. This section assembles the signals of latent demand.
1. They say it in their own words
"Methods are not shared"
Chehbouni et al. imported measurement theory into LLM evaluation and arrived at a single sentence: "technical artifacts like benchmarks and metrics are commonly shared, methodologies and practices are not ." They call for "proper mechanisms for transparent, valid and reliable evaluation" and deliver none. A position paper names the disease; it does not build the instrument. This matters because the diagnosis comes from a century-old discipline I did not invent, and it converges, independently, on exactly what Rhondda concludes: the field validates by convergent validity alone, without ever establishing reliability first.
Two traditions, same diagnosis. Chehbouni names the pathologies. Rhondda builds the registry.
"Just a starting point"
O'Brien is the paper closest to Rhondda in vocabulary: consistency as a distinct property, the i.i.d. / Bernoulli alert, the system-under-test as a composed RAG system rather than a bare model. In the sector where measurement rigor is most demanded, defense, she is the most honest about what she has not built: "With this initial framework of factors and conditions … the factors and conditions listed herein are just a starting point ." The section titled "The System" (the core of the subject) is a single paragraph. No decomposition. No granularity below "the software that wraps the LLM." The paper that frames the problem best, in the sector that demands rigor most, stops at the inventory.
"Capability ≠ reliability"
Rabanser et al. established the distinction in February 2026: capability and reliability are not the same property, and measuring one says little about the other. Khanal et al. then demonstrated it at scale a month later, on long-horizon agents , with the same core claim: "the question that matters is not whether the agent can succeed, but whether it consistently succeeds ." They built 4 metrics across 23,392 episodes and reported bootstrap CIs , a genuine rarity. But they ran each configuration only 3 times, their main novelty detector was never validated against ground truth, one of their headline metrics turned out to be redundant with the very thing it was supposed to transcend, and they never checked whether their 4 metrics were measuring 4 distinct things or the same thing four times. The field's most ambitious reliability framework remains a paper, not a substrate.
2. Five communities, one problem, zero cross-citations
Five distinct communities are working on the same problem, and none of them cite each other as doing so.
Community Representative paper What they see Defense / DoD (Test & Evaluation) O'Brien 2025 Input factors, composed system under test, consistency as distinct property NLP / LLM-as-a-Judge Yamauchi 2025 Judge design factors experimentally isolated Agents / reliability science Khanal 2026 Capability ≠ reliability, multi-metric framework Meta-benchmark / industry Benchmark Health Index (Alibaba, Feb 2026)Benchmarks as public infrastructure to audit Psychometrics / measurement theory Chehbouni 2025 Validity decomposed into 7 dimensions, conceptualization vs operationalization
O'Brien does not cite Chehbouni; Khanal cites Rabanser and discards his decomposition; the Benchmark Health Index cites neither; Yamauchi stays in the LLM-as-Judge silo. "Blind spot" is the wrong diagnosis: each community sees the problem from its own perimeter, with its own vocabulary, for its own purposes. The condition is not blindness but unassembled fragmentation : the pieces exist, and no one owns the intersection. 3. Variance: the quantity everyone suppresses
Five independent sources treat variance not as a quantity to measure but as noise to eliminate:
Source What it does Verbatim Gu et al. (LLJ survey) T =0 + majority@5 "to reduce the impact of randomness" Rabanser (agent reliability) Assumes determinism at T=0 Posited, never verified O'Brien (defense survey) Consistency figure with no values Schematic graph: no dataset, no model, no CI Zheng (MT-Bench) Single-run, no CI The foundational LLM-as-a-Judge benchmark HAL (agent leaderboard) Best-run / min-max, never CI Even on the few multi-run scaffolds available
These are the field's infrastructure, and none of them reports inter-run variance as a first-class output. The noise floor has never been characterized. Not once, by anyone. This is not an oversight. It is a structural absence. This is the hole Rhondda exists to fill.
4. The window is closing
The field is settling its conventions now (default architectures, orchestration patterns, design choices) without a substrate that qualifies their effects. Three actors are close enough to matter:
Actor Proximity Window Benchmark Health Index (Alibaba, Feb 2026)Builds a benchmark audit framework < 4 months O'Brien (Sept 2025) Names the vocabulary, stops at the inventory < 8 months Khanal (March 2026) Shares the directional thesis, lacks the warrant < 3 months
The evidence is already overabundant. What is missing is not another literature review. It is the first measurement that transforms a recognized gap into a constituted object.
Our hypothesis
The gap is real and the demand is latent — the researchers quoted above told us as much, each in their own vocabulary, without knowing the others were saying the same thing. The field needs something like Rhondda. Whether it will adopt Rhondda itself, in this exact form, is an open question and I think it is worth finding out.
What Rhondda will change and for who For system designers
You're building a system. You have a task. You face a combinatorial explosion: RAG or long-context? Single-agent or multi-agent? Self-consistency ? If yes, at what k ? You cannot test everything. You need to know which knobs are worth your time.
Rhondda shrinks that space. For your task family, the registry tells you:
Where to start. A calibrated baseline for your task family. You begin your own evals from measured defaults, not from zero. Even if you override everything, you're overriding something measured, not something assumed.Which knobs matter — and which ones move together. Some design choices dominate the outcome; others are noise. Rhondda tells you which is which, so you invest your testing budget on what actually moves the needle. It also captures interaction effects : a technique that works at one temperature may degrade at another; a model upgrade can flip the ranking between two architectural patterns. The registry measures these interdependencies and tells you their direction and magnitude. Where the returns stop. A technique that helps at one setting can be wasted compute at another. The registry captures these inflection points. You don't pay for k =9 if the benefit plateaus at k =5.In short: Rhondda absorbs the cost of rigor — controlled comparisons, noise floors, interaction effects, all measured to a shared qualification standard — and the cost of volume — the combinatorial exploration you'd otherwise have to run yourself. You inherit the conclusions, not the method.
Atlas, the free public cartography, is your interface. You navigate to your task family, consult what's known, and start your own evaluation from a pruned space instead of the full combinatorial one. You still test. You still decide. You just don't start blind.
For scientific knowledge
Rhondda has already found a blind spot in a well-established result. Wang et al. built self-consistency on a hypothesis: if an answer is consistent across multiple reasoning paths, it's probably correct. Wrong answers scatter; correct ones converge. So sample, vote, and the noise cancels out. High consistency should mean high accuracy . Rhondda's first mini-pilot — 200 items, one model, $3 — tested that hypothesis per item and found it breaks for 9% of the dataset. 18 items where the model converges to the same wrong answer, trial after trial, with near-perfect consistency. These errors are invisible to self-consistency: they look exactly like confident correct answers. The hypothesis holds on average. It fails on the cases that matter most — the ones you'd never catch without measuring consistency as a first-class output.
Rhondda changes what the field can discover. That was one model, one task, three dollars. Scale this to a registry, and here is what the field gains.
Regularities — and the variables that break them. When measurements share a common schema, you can ask questions that were unaskable before. Does this effect hold across models or is it specific to GPT? Across task families or only on reasoning? When two controlled comparisons contradict each other, the contradiction itself is a discovery: something varied that wasn't accounted for. The registry makes both the regularity and the contradiction visible, because every measurement sits in the same coordinate system.
Replication and convergence. Replication is the engine of cumulative science, but today it has no venue, no credit, and no schema in system-design research. Rhondda provides all three. A finding that survives replication becomes a building block. One that doesn't becomes a signal. The same schema also forces convergence: five communities working on the same problem in different vocabularies, unaware of each other, land in the same coordinate system. That convergence eliminates redundant effort at the scale of the field.
Coordinated effort. The registry and its operational program are public. Any team can see what has been measured, what is in progress, and where the gaps are. A lab choosing its next experiment doesn't guess — it looks at the map and goes where the frontier is thinnest. Funders can direct resources to uncovered zones . This is a shared coordinate system that makes the division of labor possible.
None of this happens by itself. It requires the active curation function — qualify, version, maintain — that Rhondda exists to carry.
For the field: what becomes possible
At the broadest level, the registry enables forms of argument, contestation, and readability that are currently impossible or extremely costly.
From conversation to argument. The field today is largely conversational: blogs, podcasts, conference talks, threads. Many have an opinion; few have qualified data. A structured substrate doesn't silence the conversation — it shifts what counts as admissible support within it. "I've observed that…" becomes distinctly weaker than "qualified measurement X shows that… under these conditions." The argumentative center of gravity shifts, without the conversation stopping.
From expert opinion to replication programs. Today, contesting an expert's claim about the efficacy of a particular design pattern has little practical recourse: you counter one opinion with another. With a registry, contestation takes the form of a replication or a new controlled comparison, qualified to the same standard. This is the classical shift from a regime of authority to a regime of evidence.
From collective opacity to readability. The field can know what it knows. Today, no one can state with clarity the empirical state of knowledge on the effects of a given design choice, on a given task type, at a given model. With a structured substrate, this state becomes readable — including its uncovered zones, which are as informative as the covered ones.
From overclaims to testable statements. Today, "multi-agent beats single-agent" or "RAG is obsolete" can be said at a conference and linger for months before anyone checks — if anyone does. With a registry, an overclaim either lands in a covered zone , where a qualified measurement already constrains it, or lands in an uncovered zone, where the registry exposes that no one has measured it. Either way, the claim becomes testable. The substrate doesn't silence opinions. It makes them checkable.
For companies
The registry doesn't just inform design. It changes the economics of building with LLMs.
Eval budget. When you don't know which knobs matter, you test everything — or you guess. Both are expensive. Rhondda prunes the search space: you test the configurations the registry hasn't already characterized, not the full combinatorial set. Your eval budget goes further.
Cost of getting it wrong. Picking the wrong architectural pattern isn't just a performance hit. It's a rebuild. You discover six months in that single-agent would have been more reliable than multi-agent for your task, and you rewrite. The registry doesn't eliminate that risk, but it shrinks it: you make the initial choice with qualified data on the table, not a collection of blog posts.
Inference spend. A technique that helps at one setting can be wasted compute at another. The registry captures where the returns stop. If self-consistency plateaus at k=10, you don't pay for k=30 — not on your own eval runs, and not in production.
Model selection. The model is a design choice like any other. The registry doesn't rank models — but it does document the measured effect of switching from one model to another within the same configuration. You can see whether the premium model actually delivers a meaningful difference for your task family before committing to its inference cost at scale.
For regulators and standards bodies
The registry doesn't mandate anything. It documents what is known. A regulator who wants to require that a system's reliability be evaluated can point to qualified measurements instead of inventing a checklist from scratch. A standards body defining what counts as evidence for a design choice can reference a public, versioned substrate rather than a consensus of expert opinion. The substrate makes evidence-based standards possible — which means lighter, more precise requirements, not heavier ones.
What the registry does not change
Five guardrails , stated explicitly to resist misreading.
1. It does not eliminate trade-offs. There is no universally superior design choice. The registry documents observed effects under characterized conditions; it does not designate an absolute winner. Trade-offs between task performance , reliability , and operational costs remain yours to make. What changes is the information level at which you make them.
2. It does not make design deterministic. Measuring an effect on a characterized task family does not guarantee the same effect will occur on your production case. Transfer uncertainty is irreducible — and the registry's qualification standard requires that measurement conditions be documented precisely so you can judge a result's relevance to your own case. The registry informs transfer; it does not guarantee it.
3. It does not turn situated effects into additive laws. Design choices interact: the effect of an architectural pattern can depend on the model, the task atomicity, the output schema , the tool ordering, the token budget, or the validators in use. A controlled comparison isolates an effect in a given experimental neighborhood; it guarantees neither consistency outside that neighborhood nor simple additivity with other separately measured effects. Rhondda's cumulativity is a discipline of localizing effects and their conditions — not a promise of simple causality.
4. It does not replace expertise. Reading the registry, judging the relevance of a measurement to your case, arbitrating between reliability dimensions, calibrating a local test — these tasks demand expertise. The registry orients expertise; it does not dispense with it. It is neither an oracle nor a substitute for judgment.
5. It must never become an automatic recommendation engine. This is its own specific risk. A registry entry does not say "choose X." It says "under these conditions, choice X produced this measured effect relative to that other choice, with this uncertainty." The slide from descriptive to prescriptive reading is not neutralized by disclaimers; it must be fought actively — through the registry's design, through the citation conventions that form around it, and through the program's public posture.
Rhondda DOES work: the $3 proof (mini-pilot) Self-consistency — sample multiple reasoning paths, take a majority vote — has been one of the most influential ideas in LLM reasoning since 2022. The core assumption: if answers agree, they're probably right. Thousands of citations later, one question had gone unasked: is the vote itself reproducible? If you run the experiment again, do the same items get the same answers?
It cost me $3 to check.
The setup In 2022, Wang et al. built self-consistency on a simple hypothesis: if an answer is consistent across multiple reasoning paths, it's probably correct. Wrong answers scatter randomly; correct ones converge. So sample, vote, and the noise cancels out. On GSM8K with PaLM-540B, they gained +17.9 points over greedy decoding — strong evidence that the hypothesis holds on average . A word on names, before we go further. Wang calls his method self-consistency because it checks whether k sampled reasoning paths agree with each other within a single run (intra-run agreement). Rhondda also uses the word consistency , but for a different measurement — inter-run reproducibility (detailed in Step 2). Don't confuse the two. Wang asks: "do the paths converge?"
This pilot asks an additional question: "is the vote itself reproducible?" — which captures something the first one cannot see.
To answer it, I changed the protocol in three ways, visible at a glance against Wang's original:
Parameter This pilot Wang et al. (2022)
Dataset GSM8K test set (200 items) GSM8K test set Model gpt-5.4-miniPaLM-540B (API deprecated in 2025) Temperature 0.7 0.7 Samples per item 30 (fixed pool)k (sampled once per item) Bootstraps per (item, k) 200 None — single majority vote k values {1, 5, 10, 20, 30} {1, …, 40} Total API calls 6,000 — Total cost $3.02 —
Each difference follows from the question:
Model. PaLM-540B's API was shut down in 2025 — replicating Wang with the original model isn't even possible. And a designer building a system today doesn't care what a 4-year-old model did anyway.Method. You can't measure whether a vote is reproducible from a single majority vote — you need its distribution. The pool-and-bootstrap approach makes the new question answerable.k range. The sweep stops at 30 because, as the next section shows, the accuracy curve simply doesn't move beyond that point. Step 1 — Establish the accuracy baseline First, I checked whether self-consistency still produces the expected logarithmic curve with a modern model. It does:
k Accuracy (mean) Gain from k=1
1 0.746 — 5 0.775 +2.9 pts 10 0.783 +3.7 pts 20 0.788 +4.2 pts 30 0.789 +4.3 pts
The gain is +4.3 points, smaller than Wang's +17.9 — expected, since gpt-5.4-mini is far more capable out of the box than PaLM-540B was, leaving less room for self-consistency to correct random errors. k=10 captures 99.2% of maximum accuracy. The 20 additional samples from k=10 to k=30 buy +0.6 points . The logarithmic decay holds.
So far, nothing new. The curve is what self-consistency predicts. But the curve is an average — and what happens inside that average is where the method breaks.
Step 2 — Measure what Wang didn't: inter-run consistency In Rhondda's vocabulary, consistency means something Wang never measured: does the system produce the same answer across repeated runs of the whole experiment ( inter-run variance )? A self-consistency vote that flips depending on which samples you draw is not consistent — even if the average accuracy looks fine.
This is not a subtle distinction. The literature routinely conflates accuracy and consistency in one of two ways: either consistency is suppressed (T=0, majority voting ) to reduce "noise," or it's treated as a proxy for correctness (Wang's hypothesis that consistent → correct). In both cases, consistency is a nuisance to eliminate or a means to an end. Rhondda treats it as a first-class measurement — because a system that is accurate on average but unpredictable per item is not reliable , and you cannot fix what you don't measure.
To measure this, for every (item, k) pair, I drew 200 independent bootstrap samples from the pool of 30 answers, computed the majority vote for each, and measured the fraction matching the modal vote. That fraction is Rhondda consistency — a per-item estimate of the vote's own reproducibility across repeated runs.
k Consistency (mean) % items > 0.95
1 0.845 46.5% 5 0.916 68.0% 10 0.940 75.0% 20 0.960 83.0% 30 0.967 84.5%
The two curves, overlaid, tell the story in one glance:
At k=10, accuracy has already captured 99.2% of its maximum — a designer looking only at the blue curve would stop here. But the red curve hasn't flattened: 25% of items still fall below 0.95 consistency . For a quarter of your predictions, the majority vote could flip if you ran the experiment again. Accuracy tells you the average score. Consistency tells you which predictions you can trust — and at k=10, one in four isn't there yet.
And the two metrics diverge as k increases:
k Spearman ρ (accuracy vs. consistency)
1 0.892 10 0.810 30 0.655
If accuracy and consistency measured the same thing, this correlation would rise, not fall, as the vote solidifies. It doesn't. At high k, accuracy and consistency become redundant for easy items and sharply divergent for hard ones — which are exactly the items you need to diagnose.
Step 3 — The four profiles Wang cannot see When you plot consistency against accuracy at k=10, the 200 items split into four populations that a single accuracy number completely collapses:
Resolved (~135 items). High accuracy, high consistency. The model converges to the right answer. Self-consistency works as intended here.Systematic bias (18 items, 9%). High consistency, zero accuracy. The model converges perfectly to the wrong answer. More samples do nothing — the vote is already unanimous. These errors are deterministic, not random, and they are invisible to accuracy alone.Ambiguous (~25 items). Low consistency, variable accuracy. Competing answers. The model genuinely hesitates between two candidates. The outcome depends on which sample you draw.Pure chance (0 items). High accuracy, low consistency. This quadrant is empty. You cannot be reliably right without being stable. Consistency is a necessary precondition for justified confidence.Five of the 18 systematically biased items:
Item Question (summary) Gold Model answer Error type
#11 "Downloads: 60, triple, -30%..." 366 60 (24/30 draws) Miscalculated reduction step #13 "Lemon tree: $90, $7.50/year..." 13 12 (27/30) Off-by-one in payoff logic #129 "Watermelons, peppers, oranges..." 880 15 (30/30) Fundamental comprehension failure #145 "Housekeeping profit..." 20 92 (30/30) Systematically wrong calculation #194 "Neighborhood prank..." 24 6 (30/30) Misinterpreted problem structure
These are not unlucky draws. The model is reproducibly wrong — trial after trial, with unanimous convergence. This is Wang's central hypothesis breaking, item by item: the answer is consistent across reasoning paths, and it is not correct. Consistency did not imply correctness. The hypothesis holds on average — that's why the mean accuracy curve looks fine. It fails on the cases that matter most — the ones you'd never catch without measuring consistency as a first-class output.
Step 4 — More samples ≠ better answers Self-consistency's implicit contract is monotonic: more samples → better accuracy. The pilot shows this contract is violated in both directions.
11 items (5.5%) change their majority vote between k=5 and k=20:
Item Vote at k=5 Vote at k=20 Gold Outcome
#37 30 75 75 ✅ Corrected #87 4 22 22 ✅ Corrected #136 15 40 40 ✅ Corrected #163 72 92 92 ✅ Corrected #173 19 51 51 ✅ Corrected #32 80 100 80 ❌ Degraded #57 3 42 3 ❌ Degraded #175 95 35 95 ❌ Degraded #74 765 150 255 ❌ Remained wrong #128 70 50 120 ❌ Remained wrong #162 8 13 32 ❌ Remained wrong
Five items improved. Six degraded or stayed wrong. The mean accuracy still drifts up because corrections slightly outweigh degradations in magnitude — but you don't ship the mean. You ship individual predictions, and for those 6 items, spending more on inference made the answer worse . Without per-item consistency, you cannot tell which items you're improving and which you're degrading. You're betting blind.
What the pilot proves — and what it doesn't This is a minimum viable demonstration. $3, one model, one task family, one temperature . It shows the method surfaces structure that accuracy alone misses. It does not deliver the registry — and it does not claim to.
The pilot shows Only the full program can establish
Consistency captures information that accuracy misses — on one model (gpt-5.4-mini). Whether this holds across 16+ models from different lineages, scales, and reasoning tiers — and whether each model has a distinct consistency signature. The quadrant analysis reveals four distinct item populations — on one task family (GSM8K arithmetic reasoning). Whether the four profiles generalize across multiple task families with different verification modes, response spaces, and reasoning depths. 18 items (9%) are systematically biased at T=0.7 . How consistency varies with temperature across a full sweep (T ∈ {0.3, 0.5, 0.7, 0.9, 1.0}) — the consistency-diversity frontier is entirely uncharacterized. At k=10, 99.2% of maximum accuracy is already captured — for one design factor (candidate count) in one regime (single-call CoT). Where the inflection points fall across the 272 design factors and their interaction effects Consistency can be measured black-box logits , seeds, or model internals. What white-box Inter-run consistency can be quantified per item — on one reliability dimension . Extending the method to robustness predictability safety 200 items surface a 9% systematic bias rate. Full test sets (1,319 items for GSM8K alone) to estimate population-level rates with confidence intervals and identify item-level predictors of bias.A bootstrap consistency metric can be computed and reported. Consistency as a qualified, versioned registry entry with documented experimental conditions, quantified uncertainty, and a composable schema.
What this changes for a designer — today Even at $3 and 200 items, the pilot surfaces three things that no existing evaluation framework provides. They are diagnostics, not prescriptions — their validity beyond this specific setup remains to be tested.
1. A way to separate systematic from random error. On this model and task, low consistency + low accuracy corresponds to items where the model is guessing; high consistency + low accuracy corresponds to items where it is reproducibly wrong. Accuracy alone cannot tell these apart. If this distinction holds across models and tasks, it would change how designers allocate their debugging effort: random errors might respond to more samples or a different decoding strategy; systematic errors likely require a different model or task decomposition.
2. A per-item confidence score — without logits. For any black-box API, consistency gives you an interpretable number: "this prediction's vote would remain the same X% of the time if I reran the experiment." The pilot demonstrates this is computable from a pool of 30 answers with bootstraps. Whether the 0.95 threshold is the right one, and whether routing low-consistency items to a stronger model actually improves outcomes, are open questions the pilot does not answer.
3. A signal for when to stop spending on inference. For this model and task, k=10 captures 99.2% of maximum accuracy and 97.2% of maximum consistency. The consistency curve tells you whether the plateau is real or whether individual items are still oscillating underneath a flat average. Without it, you're deciding when to stop based on accuracy alone — and as the quadrant plot shows, items with identical accuracy can have radically different consistency.
Conclusion $3, one model, one task, one design factor. The pilot surfaced a 9% blind spot in self-consistency — errors invisible to accuracy, unfixable by more samples. It may be larger on other models, smaller at other temperatures. The pilot cannot say.
What it can say: the method works. Per-item consistency, black-box, quantified uncertainty — measurable for the cost of a coffee.
It gives a lower bound. If $3 finds a 9% hole in a technique the field has trusted since 2022, the expected yield of 272 factors × 16+ models × multiple task families is not zero. Most combinations will be empty. Some won't be. No one knows which is which — because no one carries the function of measuring it.
That's the argument. Not that $3 solved anything. That $3 proved the signal exists — and if $3 can find a 9% blind spot in a technique the field has trusted for four years, imagine what a full program will uncover about designing AI systems.
How I built it Building a coordinate system for a field that doesn't have one is not a theoretical exercise. It is a methodological puzzle. This section recounts how I solved it.
I built it ab initio , without any community or institution, assisted by two LLMs in disjoint roles: Claude Opus 4.5 to produce, and GPT-5.3 to ruthlessly attack. Yet, every final decision was mine, and every choice is documented alongside its discarded alternative. Without an external arbiter, I base my legitimacy on transparency.
But that arbiter will come: the methodology will be the subject of a peer-reviewed article. Here, the account remains deliberately narrative so as not to overwhelm the reader and to prioritize readability.
Why ab initio ? I would have gladly adopted an existing structure: rebuilding ab initio held no appeal; it is slow, exposed, and one repeats mistakes others have already corrected. But such a structure would have had to exist first.
The raw material, however, was not lacking, and I want to be clear: claiming to start from an empty field would be dishonest. Mappings were abundant, from LLM agent surveys to RAG taxonomies, from inventories of prompting techniques to catalogs of agentic patterns (reflection, tool use, planning, multi-agent), all the way to the already established framing of " compound AI systems ".
I had assessed the landscape through two parallel research efforts:
An in-depth one : cross-referencing three models from different families (GPT-5.3, Claude Opus 4.5, and Gemini Pro) to triangulate academic literature and technical articles.A direct one : as exhaustive as possible, on databases like Semantic Scholar (which kindly provided me with an API key, despite this access being normally reserved for academic institutions, nonprofits, and government entities).A formal systematic review would require an article of its own; that is not the purpose here, but this dual census is sufficient to support what follows.
I have, in fact, borrowed without hesitation from what I evaluated as the most solid existing foundations:
From the tradition of Fisher's experimental design : the very vocabulary of factor, level, and attribution. From HELM : the posture of a public registry and standardized schema. From Rabanser et al. : the four-dimensional vocabulary of reliability . From Kapoor's HAL : the cost-aware discipline. Yet none of them offered me the unit of analysis I needed: all of them characterize models or agents, whereas I was looking for the effect of a system design choice with controlled variables. More of this positioning in the Where Rhondda fits in the literature section. These mappings could not serve as a load-bearing foundation either: each relied on an organizational principle that my grid exists precisely to reject. Some were goal-oriented: sorted by capability, task, or metric ("techniques that improve reasoning", "what reduces hallucination"), and thus indexed on their effects rather than the degrees of freedom that produce them. Others were indexed on implementation: RAG patterns, families of agent frameworks—that is, specific mechanisms elevated to categories. Still others were heterogeneous in granularity, mixing a component, a technique, and an architectural principle in the same list. And almost all lacked a comprehensive structure: they were just open-ended lists, never an exhaustive, non-overlapping breakdown of the entire request flow, and nothing separating what can be taken for granted before measurement from what cannot.
Relying on any of them would therefore not have saved any work: it would have imported the flaw that the grid was meant to correct. A goal-oriented foundation would have perpetuated the opportunistic taxonomy; an implementation-indexed foundation would have frozen mechanisms instead of levers; a foundation with heterogeneous granularity would have propagated its fuzziness to all levels. To inherit, here, was to inherit the disease.
This is why the ab initio approach was not a choice, but an imposed condition. This very absence is the project's raison d'être: had a proper coordinate system already existed, the Rhondda grid would be unnecessary. I built it precisely because this foundation was missing.
Decomposing the space: From a goal-oriented false start to the six structural families The ultimate goal of Rhondda is to serve as a universal coordinate system: a framework where any LLM-based system, from a simple API call to a complex multi-agent architecture, can be precisely mapped as a combination of structural choices.
But to achieve this, the grid first had to be built. How should this vast space of possibilities be carved out? This is where the most structuring reversal of the entire project occurred.
How to divide the space of design choices ? My first instinct was goal-oriented: for every performance criterion or reliability dimension I already knew I would eventually have to measure, I asked which design choices affected it. The hostile review (the adversarial pass I ran on every draft) demolished this approach, and rightly so. A goal-oriented construction creates an opportunistic taxonomy: it groups things by their intended effect rather than by actual design choices.
For example, if we categorize "Retrieval-Augmented Generation" ( RAG ) simply under "Choices that improve task performance " (a performance criterion), we ignore the fact that structurally, RAG is just an external data fetch that modifies the input context. It is a mechanical choice that also drastically impacts Robustness and Safety (reliability dimensions), not to mention latency and cost. A lever is just a lever; what it moves is a separate question.
This goal-oriented approach guarantees overlaps and blind spots, and forgets that a design lever exists independently of the dimension it is intended to move. I therefore established a foundational rule:
The grid must remain structural, not goal-oriented.
This correction went against my initial idea, but it remains one of the decisions I am most certain of. From then on, looking at goals would only serve as a completeness check, never as a construction principle.
Structural according to what invariant? Whether it is a simple API call or a multi-agent setup, every system transforms a request into a response through at least one model inference. This sequence—the flow of a request—is the only invariant shared by all architectures. I applied a foundational question to this flow at the highest level:
In an LLM-based AI system, where does something happen that could be built differently?
The answer is not a hypothesis, but a logical decomposition of the minimal flow. Since an LLM-based system calls a model at least once, we can map the functions surrounding this inference. Three are present from the very first call, and a fourth governs their sequence:
Input and Context Construction (A) : There must be something to infer.Inference (C) : The core operation itself.Output Processing (D) : Handling what the model produces.Orchestration (E): The sequence of the flow. Trivial for a single call , but a major source of variation as soon as there are multiple inferences (e.g., sequential chaining, conditional routing, loops, or parallel execution). It is always present since every flow has an order. There are only two ways a pure inference core can be augmented: extending it into space (stepping outside the model) or into time (persisting beyond the call). "Elsewhere" and "after" exhaust what can be added. No other structural extension is possible without falling back into the existing functions:
Interaction (B) : Reaching out into the external world.Persistence (F) : Storing state beyond the session.Families B and F appear conditionally; the other four are strictly necessary. These six functions are the families. I chose this term because it is neutral and avoids clashing with existing academic vocabulary.
The partition is therefore logically derived, not arbitrarily assumed. What remains is to test its robustness. This means verifying two things: first, that an independent researcher asking the same question would arrive at the exact same six families. Second, that highly unusual or complex AI architectures can be entirely mapped into these categories without leaving any missing pieces. These tests will confirm the grid's structural integrity.
Do all families apply to every system? No. For instance, on a single API call, reaching out to the world and persisting a state are absent or trivial. To avoid falsely suggesting that everything applies everywhere, I assigned each element an applicability value:
Generic : Applies to almost any LLM-based system.Conditional : Applies only if a specific architectural feature (like external interaction or persistence) is present.Specialized : Applies only to highly specific architectures or use cases.And the model itself, where is it in the grid? It is neither an ordinary component nor completely off-grid. Depending on what is being measured, the model assumes one of four distinct roles:
A fixed condition : Held constant to isolate the effect of a specific design choice.A stratification variable : Used to group results by model capability.The lever being varied : The actual variable being changed to study its direct effect.A source of interaction : A factor whose effect changes when combined with other design choices.I therefore assigned the model a variable status. However, the third role—the specific design lever being varied to study its effect—did not yet have a formal name. Addressing this gap was the next critical step.
Choosing the granularity: From broad families to atomic design factors Naming the atomic unit Once I mapped the architectural space into six broad structural families, I hit a fundamental problem.
Could these six families serve as the actual unit of measurement? No. The whole point of Rhondda is to isolate and measure specific effects. To do that reliably, you have to drill down to the most atomic level possible. If a unit is composite, you'll never know which of its sub-parts actually caused the effect you're seeing. For measurements to be comparable and stackable over time, they must apply to a single, indivisible lever that is defined exactly the same way every time. We needed a stable, shared atomic unit of attribution.
I called this unit a " design factor ". It plays on a double meaning between system design and experimental design, which was exactly what I wanted. It bridges these two worlds, accommodating both fine-grained settings and massive structural choices.
The structural zone of variation But could I just fill the six families directly with atomic design factors ? The leap was simply too big. Jumping straight to the atomic level risked three massive drifts:
Upward into broad components : If the unit is too large (like "the entire retrieval module"), it isn't a specific lever you can test. It becomes a whole sub-system, which collapses right back into the broad macro-families we were trying to break down.Downward into atomic design factors : Which meant rushing the taxonomy before the structure was ready.Sideways into implementational composites : Which meant anchoring the grid on concrete artifacts rather than abstract functions.To avoid these pitfalls, I needed a middle layer. I called it the structural zone of variation.
Where does the grid stop and the taxonomy take over? This middle layer created a strict boundary between two very distinct concepts:
The Grid : The stable architectural skeleton, made up of families and zones . The Taxonomy : The exhaustive catalog of atomic design factors that populates that skeleton.The grid stops at the zone. Descending to the atomic level is the taxonomy's job.
But why wait? An item like " self-consistency " alone hides at least five sub-factors (like the number of samples, decoding temperature , prompt structure, aggregation mechanism, and tie-breaking rules). The temptation to dig into that atomic basement right away was incredibly strong. But I held back to avoid two disasters:
Cascading rework : The skeleton dictates everything else. If you place atomic factors on a skeleton that is still moving, you guarantee yourself endless corrections.Absence of a referent : You can't judge if a factor is truly atomic unless you know its exact boundaries, and those boundaries are defined by the overarching partition. Without a stable framework, "atomicity" is just a buzzword.This forced a strict discipline onto the entire process:
Never mix three operations: defining the families, listing the design factors, and judging their atomicity.
Mixing them blurs the granularity and spreads errors everywhere. This rule was absolute from day one.
Should we model the recursivity of zones? Think about a retrieval system relying on a previous retrieval, or a shell starting a container that runs a browser. The nesting is real. Unfolding this infinite loop would have exploded the map. But ignoring it meant missing a deep structural fact. So I opted for a compromise between a "flat" grid that lies by omission, and an "explicitly recursive" grid that overcomplicates everything:
Flag recursivity without unfolding it.
Should these methodological rules be locked early, or adjusted along the way? If a foundational rule drifts halfway through the project, it corrupts everything downstream. That's why I explicitly froze the core principles of the grid (the granularity, the applicability logic, the model's variable status, recursivity, and depth limits) before I started populating a single family with data. I reinforced this with a strict discipline of order:
The overarching structure (the families) must be completely frozen before digging into the sub-levels (the zones). If you get the granularity wrong when filling out the first family, you'll just reproduce that error on the next five.
Which meant I had one immediate task left: brutally testing and locking those six families before taking a single step downward.
Testing and locking the families Does the partition actually hold up? Drafting a partition is one thing; validating it is another. Before I let myself dig into the sub-levels, I ran the six families through a brutal coverage test across every architecture I could think of: single API calls , chain-of-thought , zero-loop RAG , rigid LLM workflows, tool-calling agents , ReAct loops , and full multi-agent systems.
Every single architecture broke down neatly into a subset of those six families. There were no leftovers. Nothing fell through the cracks. There was absolutely no need for a seventh function.
The hostile review and the blind spot Then, I started attacking my own grid.
The most useful critique hit Family B. I had originally named it something far too narrow: "Acquisition of external information". My hostile review process immediately flagged a massive blind spot: mutative actions (like sending an email, triggering a payment, or deleting a file).
These are very real design choices that carry huge safety risks, and calling it "information acquisition" excluded them entirely. A system doesn't just read the world; it acts on it.
I immediately renamed Family B to "External Interaction". The hostile review exposed the flaw, but the structural logic fixed it.
Where do "Agents" fit in? I also had to make a definitive ruling on a major industry buzzword, and my decision held up under intense scrutiny:
No family is dedicated to "the agent". An agent isn't a new degree of freedom; it's just a specific composition of the six families.
Giving "agents" their own family would have completely fractured the grid. I ruled that Orchestration (Family E), when combined with Context, Inference, Output, and Persistence, was more than enough to perfectly capture any agentic behavior.
The final top-level skeleton After a few more minor boundary corrections, the top-level structure was officially locked. I had my six foundational families:
A: Input and Context Construction (Generic)B: External Interaction (Conditional)C: Inference (Generic)D: Output Processing (Generic)E: Orchestration (Generic)F: Persistence (Conditional)The overarching skeleton was finally frozen. I was ready to descend into the structural zones of variation.
Emerging the structural zones The existence test and the golden rule With the families locked, I descended into the structural zones . I went through six brutal production-and-review cycles. Almost every well-defined zone in Rhondda was forged by correcting an initial mistake.
To prevent myself from creating arbitrary boundaries, I set a strict existence test:
Can this function be built in multiple different ways? If the answer is no—if it's just a single, isolated mechanism—then it isn't a zone. It's an atomic design factor , and it must be pushed down to the taxonomy.
I paired this with a golden rule for classification:
Always classify by primary function, not by implementation.
Two completely different mechanisms can perform the exact same function. For example, you can filter toxic outputs using a simple regex script, a dedicated moderation API, or a secondary LLM acting as a judge. Those are three entirely different implementations, but they all serve the exact same function. Grouping them by their actual function ensures the zone stays completely stable, even when the underlying code or implementation changes.
Discovering composites and fixing blind spots During these six cycles, I hit a snag: the strict family partition couldn't house everything cleanly. I had to recognize a distinct, recurring architectural pattern: composite zones.
These are functions that naturally span across multiple families at once. For example:
Multi-candidates spans Inference (C), Output Processing (D), and Orchestration (E). Retry loops span Output Processing (D) and Orchestration (E).Routing spans Inference (C) and Orchestration (E).Deliberation spans Orchestration (E) along with almost everything else (A/C/D/F).Forcing these multi-family workflows into a single family just for the sake of symmetry would have completely skewed the attribution of effects. By explicitly naming them as a distinct class of objects—rather than trying to shoehorn them into standard zones—they became measurable and manageable.
I also had to make several other massive corrections:
Family A (Context) exploded from six zones to eleven, as hostile reviews kept exposing blind spots (task representation, message structure, tool exposure). I introduced transversal attributes for things that affect every zone in a family but aren't zones themselves (like access regimes or fault models for Family B). I pulled non-LLM auxiliary models (like rerankers or OCR) completely out of Family C (Inference). Family C is strictly for LLM inference; forcing an embedding model into it was a categorical error. Knowing when to stop After six grueling cycles, the structure finally stopped moving. The blind spots had shrunk from massive structural gaps to tiny boundary tweaks. No seventh family emerged. No new zones were needed.
This gave me my stopping criterion: the process had officially converged.
However, I intentionally declared the grid "provisionally stabilized" rather than permanently frozen, sticking to a core principle:
Never definitively freeze a structure that might be forced to move by downstream discoveries.
The result: Structural Grid v0 At the end of this phase, I held the Structural Grid v0 in my hands.
It contained six families and their respective zones (11 for Context, 6 for Interaction, 5 for Inference, 5 for Output, 6 for Orchestration, and 5 for Persistence). It was provisionally stabilized, built entirely before any descent into atomic factors, and totally pre-empirical.
I had laid down an empty coordinate system. Nothing was hanging on it yet—and at this stage, that was exactly what I wanted.
Surfacing the design factors: the proto-taxonomy With the genesis complete, I finally descended to the atomic design factor level, family by family. This wasn't a quick theoretical exercise. It was a massive, cognitively grueling undertaking that required manually mapping the entire technical landscape of modern LLM architectures.
This pass was massively doctrinal. I was no longer cutting up the space; instead, I was laying down the strict rules that would make all future empirical measurements interpretable.
It was during this exhausting phase that the attribution framework emerged. This framework is the complete logical toolbox that allows you to confidently say:
This specific change in performance is caused by this exact design factor, and nothing else.
Without it, true measurement is impossible.
1. Defining what makes a "Design Factor" To prevent the taxonomy from being flooded with fake design factors , I had to define strict entry criteria. First, I reused the three criteria that had guided the genesis (manipulability, attribution, and recurrence) as a mandatory admission test.
But to be officially promoted into a fully autonomous design factor, a candidate had to pass a much harder test. I created a rigid, frozen documentation template (finalized after a pilot run on Families C and D to ensure comparability). I hardened this into a strict conjunctive test:
If a candidate cannot clearly define its own parameters, preconditions, couplings , edge cases, and plausible sources, promotion is immediately blocked.
I explicitly rejected a disjunctive test ("three out of five is good enough"), which would have allowed weak, non-measurable factors to slip through.
2. Isolating the variables Even if a factor passed the promotion test, it was useless if it couldn't be isolated. Interdependence exists everywhere in an architecture, but grouping all dependencies under a single blanket term like "coupled" was masking critical methodological differences.
If you want to track attribution properly, you need to know exactly how two factors are tied together. I split " coupling " into three distinct situations that ruin isolated measurement:
Intra-system co-variation : Varying this factor automatically causes another design factor to vary along with it (e.g., enforcing a strict JSON schema output almost always forces you to simultaneously add a new explicit instruction in the prompt).Applicability constraints : This factor is physically unavailable under certain providers or runtimes. If you want to test it, you are forced to change your provider, which destroys the isolation of the variable because you are modifying two things at the same time (e.g., testing the min_p sampling parameter forces you to switch to an open-source runtime because commercial APIs don't expose it).Out-of-scope risk : Changing this factor fundamentally alters the underlying task itself. If the task changes, you are no longer measuring the system's architectural performance (e.g., altering an operational rule so much that it triggers a completely different fallback workflow; you are now measuring the fallback task, not the original rule).With interdependence properly untangled, I locked down four strict statuses of atomicity: atomic, coupled, composite, and transversal attribute. The composite status is particularly dangerous: I established that any effect attributed to a composite factor applies to the block as a whole, never to its individual constituents.
3. Building the measurement tools With the factors defined and isolated, I built the specific tools required to actually conduct measurements without cheating:
The equivalence clause : You can't compare apples to oranges. Some design factors can only be measured if you prove the baseline is equivalent (e.g., comparing two different prompt structures is only valid if you can prove they contain the exact same baseline information).Attributional ceiling vs. effective downgrade : A factor's theoretical validity is fixed in its definition. If an experiment is poorly controlled (for example, measuring a factor using a closed-source LLM with hidden internal parameters), the measurement itself is downgraded in reliability , but the permanent definition of the design factor remains untouched. Observability regimes : Not all levers are equally transparent. I had to grade every factor across five strict levels of observability (e.g., changing a temperature parameter is directly observable in the code, whereas a provider's hidden internal routing mechanism is completely invisible and can only be inferred). Precondition load : How much technical setup is required before you can even test a specific lever? I graded this across four levels (e.g., tweaking a prompt has a 'low' load, but testing a RAG pipeline has a 'high' load). I actually invented this grading system after I had already finished mapping Family A. Instead of retroactively auditing Family A, which would have taken an unreasonable amount of time, I applied it forward-only (only to subsequent families). I deliberately pushed the resulting inconsistencies into my technical debt register. 4. The reality of mapping: Rulings and technical debt Applying this framework to the entire state of the art forced several hard architectural rulings:
Transversal motifs : The mapping surfaced five recurring design motifs (e.g., the exposure channel, order/packing). I explicitly treated these as non-canonical and non-retroactive: they were patterns to watch out for, but they did not override the structural breakdown.Blocking a merger : An audit proposed merging four specific inference parameters into two. I refused, because the merger would have mixed a syntactic object (like JSON formatting) with an operational regime (like the strictness of the constraint), creating nonsensical cells.Ruthless rejection : I explicitly denied promotion to a candidate related to provider-side routing (where the LLM provider opaquely routes requests between internal models) because its plausible sources weren't discriminative enough, causing it to fail the conjunctive test.The empty placeholder : Most importantly, I treated an entire subsection of the taxonomy, initially populated with thirteen interaction-related factors, as an empty doctrinal placeholder. These thirteen candidates had been originally promoted simply because they appeared together frequently in the literature. But upon strict audit, not a single one passed the conjunctive test. Rather than artificially filling the register just for the sake of symmetry, I kept it visibly empty:An honestly empty register is better than a falsely full one.
Finally, the order in which I mapped the taxonomy wasn't random: Inference before Output Evaluation, then Orchestration , etc. I started with Inference because without it, my foundational thesis ("system design is not model selection") relied on a visible absence. Inference was doctrinally mature and unlocked the evaluation metrics, which depended entirely on it.
Making the grid operational The final step was to assemble my local drafts into a single, canonical v0.1 grid. Up to this point, the taxonomy was just a collection of loose notes. Canonization was the process of turning this theoretical catalog into an operational measurement tool. Without freezing and structuring these drafts into a database, we could not run consistent experiments: the definitions of what we are testing would shift over time, and we would have no way to reliably compare results. It is the step where we build and calibrate the thermometer before taking the temperature . Unlike the previous exploratory phases, I planned this assembly strictly ex ante across nine logical phases, where each step was dictated by structural dependencies.
1. The guiding principle The entire assembly process was guided by a core rule that I established on day one:
A design factor can be finalized before measurement, but the attribution of an effect cannot.
In other words, I could freeze the structure of the system (the categories, the design factors , and the boundaries between zones ), but I was strictly forbidden from guessing how much each design factor actually impacted performance before running experiments.
I separated the work into two distinct stages: carving the individual building blocks (proto-construction) and then cementing them together (canonization). To make sure the foundation did not shift under my feet, I locked the draft files. Once locked, no sneaky or undocumented changes were allowed; any edit required updating the file security hash and documenting the exact impact of the change. Without this discipline, a long-running project risks silently relying on versions of the family drafts that have shifted.
2. The nine phases of assembly The order of the nine phases was not arbitrary. It followed a sequence dictated entirely by structural dependencies:
Phase 0 and Phase 0.5 (Setup and Guidelines) : Before any assembly began, I locked down the files and established the rules. In Phase 0, I took the existing rough drafts of my taxonomy (the initial lists of design factors I had drawn up for families like Context and Interaction) and froze them, along with the academic and industry papers I was referencing. Freezing these files ensured my raw material remained completely stable throughout the process. In Phase 0.5, I defined the standard formatting rules. This included creating the templates for the catalog cards, setting up automated validation scripts to catch typos and inconsistent categories, and writing the guidelines for when to stop auditing. Phase 1 (Locking down Context and Inputs) : I finalized the first family , which covers how inputs and instructions are fed to the model. Because all other parts of the system depend on this initial structure, it had to be sealed first to serve as a stable foundation for the rest of the template. This resulted in 91 design factors. Phase 2 (Testing the Template) : I ran a pilot test using a very clean section (Output Processing, which had no technical debt) and a more complex section (Model Inference) to test my card template. This allowed me to verify if the template worked on both straightforward and highly complicated parts of the system.Phase 3 (Converting User-System Interaction) : I formatted the section on user-interaction (59 design factors) into the template. During this phase, I kept the template design strictly frozen to avoid tweaking it on the fly, which would have ruined the consistency . Phase 4 (Integrating the Rest) : I converted and integrated the remaining sections: Model Inference, Output Evaluation, Orchestration (how steps are chained), and Persistence (how memory is saved). This brought the total count of design factors in these sections to 122. Phase 5 (Assembling the Catalog) : I compiled all six families into a single master document, ran a thorough check for duplicate terms, mapped how they overlapped, and listed any technical debt (questions to resolve later).Phase 6 (The Adversarial Review) : I ran a final, aggressive audit across the entire system using a separate LLM to challenge my assumptions and check for contradictions. The audit ended with zero critical conflicts.Phase 7 (Building the Database) : I converted the text document into a structured, machine-readable database (the global registry) and added placeholders to hook in future experimental data.Phase 8 (Setting the Rules for Future Tests) : I wrote the exact criteria that any future experiment must meet to be considered valid. This ensured the testing protocols were fully defined before a single test was run.3. The three controls To keep the assembly process on track, I set up three strict control mechanisms:
The audit stop rule : To avoid editing forever (the "perpetual draft" syndrome), I forced myself to stop auditing as soon as two consecutive reviews revealed no new structural conflicts. Any minor remaining notes (like phrasing tweaks or future questions) were simply tagged and shelved for later, rather than letting the debate drag on.The automated validator : All classification statuses are guarded by a strict machine script. A category or status cannot be invented on the fly; if a new status is needed, I am forced to update the database schema formally rather than slipping in undocumented labels.Strict versioning rules : The version numbers directly show where the project stands. The v0.1 is the first structural layout, v0.2 will integrate feedback from external theoretical audits, and any later version suffix will mark changes that come from real-world measurements.A v1.0 release is strictly forbidden until the grid has gone through a full round of real-world testing.
4. The final numbers The final stats of this assembly process were as follows:
272 design factors (91 for Context, 59 for Interaction, 22 for Inference, 26 for Output Processing, 36 for Orchestration, and 38 for Persistence).1 design factor absorbed into another , 9 non-promotions (design factors whose contents were distributed into other cards), and 1 doctrinal placeholder kept empty for logical symmetry.32 technical debts (unresolved questions or formatting cleanups) logged at the v0.1 freeze. These were not resolved immediately because the vast majority could not be settled on paper. Instead, they waited for the first empirical measurements (such as testing whether API parameters actually behaved as declared in practice) or were permanent tracking rules to be applied to every future experiment.0 validated effects (no guesses on which choices work best).This last number was the key to the entire project. Zero validated effects meant I had built a thermometer, not taken a temperature. I created a coordinate system to map possible design choices, frozen before anyone started measuring their actual impact. That was what pre-empirical meant.
The grid was built. The taxonomy was documented. This is exactly where the project stands today. Now the real work begins: using it to measure, accumulate, and transform how we design AI systems.
Auditing my own methodology Auditing my own methodology is not a formality. If Rhondda is to fix the problem of non‑cumulative knowledge, its construction must itself be falsifiable. An elegant taxonomy that nobody can scrutinize would be the worst outcome. So I made sure I traced every major decision back to its justification – structural, doctrinal, or unresolved – and here is what I want to bring to the reader's attention.
Taking stock of the arguments I took stock of the justifications I had relied on throughout the entire genesis of the grid.
The result was extremely clear. Almost every single decision I made to establish this grid was backed by a recorded, substantive argument. These arguments fell into one of two categories:
Structural : Based on degrees of freedom or the logical partitioning of the workflow.Doctrinal : Based on a strict principle I had established beforehand.The stopping criterion (the point where I declared convergence) was the only decision based purely on process rather than a structural argument. I flagged it explicitly because this exact level of transparency was what gave the trace its value.
Disarming common misunderstandings There were two important nuances I needed to clarify to avoid common misunderstandings.
First, many of my best decisions were discovered by the hostile review process, but they remained justified by the underlying structure . The renaming of Family B, the massive expansion of Family A, and the eviction of non-LLM models from Family C were all triggered by hostile audits. However, confusing how an idea was found with what actually justified it would have been a classic mistake. The structural argument was always there; the review simply brought it to light.
Second, the foundational question I asked at the very beginning—"Where in the workflow could something be built differently?"—didn't just casually generate the partition. When applied to the minimal workflow, it actively derived it through exhaustive logical decomposition. Every function is accounted for, and the resulting grid is logically consistent . It isn't a hypothesis; it is a structural necessity.
However, while the methodology supports logical robustness , a theoretical coordinate system isn't the end goal. Only strict empirical measurements can definitively validate it in the real world . Proving the ultimate resilience of this structure (through independent reproduction and adversarial architectures) is the next required step. This belongs to the empirical measurement phase of the program—which is exactly the phase I am asking the grant for.
How I used LLMs to stress-test the grid Orchestrating the workflow I didn't build this bare-handed. I orchestrated two frontier language models from entirely different LLM families, assigning them strictly disjoint roles. This separation of model families was critical: it ensured that the attacking model didn't share the same blind spots as the producing model.
Claude Opus 4.5 was the Producer : Generating the structural families, zones , and design factors from scratch based on my instructions and framework. GPT 5.3 was the Attacker : Subjecting every single draft to a hostile review, hunting for flaws.I was the Arbitrator : The third role, and the only one allowed to make decisions.Grounding the knowledge To be clear, this Claude/GPT duo was only used for the downstream production phase of the grid itself. The upstream phase (mapping the existing academic literature) relied on a much wider net: triangulation across three distinct LLM families (Claude Opus 4.5, GPT 5.3, and Gemini Pro) anchored by Semantic Scholar as a non-generative, hard-data source.
My domain knowledge didn't just come from what two models "knew" about a topic. It was grounded in an independently indexed documentary base, cross-examined by three distinct AI families. This upstream heterogeneity helped mitigate the risk of circular LLM bias during the downstream production.
The hostility charter The workflow was strictly maintained for every single structural family : production, self-critique, hostile review by the adverse model, my triage of the critiques (adopt, nuance, or reject), patching, and a final integrity check. This took three to four rounds per structural family.
I gave the reviewing model an explicit charter, and I held it to that standard from start to finish:
Posture : Hostile and uncompromising. The instruction was "Find the blind spots; do not validate the text."Targets : Overlaps, fake design factors, over-absorption, poorly drawn boundaries, garbage-can zones, and internal contradictions.Integration threshold : A critique was only retained if it closed a legitimate gap.Above all, I enforced an unbreakable rule of governance:
The adversary exists to detect, never to decide. The final decision is always mine.
I also formalized a clear stopping criterion: halt the cycle after two consecutive rounds yielded no new doctrinal blockages. I refused to run the reviews into rhetorical exhaustion.
The signature of the protocol and the reliability of the trace This protocol has a specific signature, which I fully own. The cycles were strictly sequential (each patch influenced the next). The adversary was singular (one specific hostile perspective). This rigorous review process produced a measurable level of robustness : by the end of the cycles, no fundamental disagreements survived across the six structural families, signaling that the cuts had properly hardened.
(Generating genuinely alternative architectures is a separate task, one I am formally designating as a future extension: a clean-room adversarial pass, with no patch history, conducted by a third LLM family tasked exclusively with proposing different cuts.)
Finally, a word on the reliability of the trace. Every single step relied on a strict triangulation between three positions: my arbitration, the producer, and the reviewer. Because no decision was ever single-sourced, this redundancy protected the content. Anything advanced during production was immediately stress-tested in review and frozen in the artifacts. That systematic triangulation is what makes the methodological trace solid.
Where is Rhondda right now and what's next Rhondda sits at a deliberate inflection point: the pre-empirical scaffolding is complete, the method has been proven on a minimal pilot, and the program is now seeking the funding to execute its first real test. The taxonomy is a structured hypothesis about decomposition — its validity is unknown, and empirically stress-testing it is a primary object of Year 1 falsification. Funding enables the first measurement cycle; a populated registry is the output of that validation, not its premise. Here is exactly what exists today, and what the next phase would build.
Where it is today What exists 1. System Design Grid. • v0.1, frozen, machine-readable — 272 atomic design factors across 6 families and 38 zones . • ~959 couplings mapped with polarity and observability status. • 0 validated effects (by design — the grid is a thermometer, not a temperature ). • 32 technical debts logged at freeze. • Every methodological choice documented with peer review already in view.
2. Controlled-comparison protocol & measurement framework. • Isolation discipline: a performance change is attributed to a design factor only if configurations differ by that single variable. • Two non-substitutable core properties: task performance + 4 reliability dimensions ( consistency , robustness , predictability , safety ). • 12 task attributes defining the Discrete Verifiable Sub-Space as the zero-noise baseline. • Factorial designs for interaction effects , mandatory uncertainty quantification. • Execution metadata (latency, tokens, cost) tracked on every run.
3. Operational measurement programs. • Cycle 0 ( single-call LLM) — fully specified, ready to execute upon funding. • Cycle 1 ( LLM-as-Judge ) — designed, sequenced after Cycle 0.
4. Atlas specification. • Coordinates View: interactive 2D matrix + couplings interaction layer (local network on hover, configuration landscape on click). • Designers View: practitioner gallery with dynamic facets, Active Workspace, empirical panels. • Static mockups exist for both views.
6. The $3 pilot. • 200 GSM8K items, gpt-5.4-mini, k ∈ {1,5,10,20,30}, 6,000 API calls , total cost $3.02. • Proved per-item consistency (measured black-box via bootstrap) surfaces structure that accuracy alone collapses. • Found a 9% systematic bias blind spot in self-consistency — errors invisible to accuracy, unfixable by more samples.
What does NOT exist • No qualified registry entries — the pilot demonstrated the method but wasn't run under the full qualification protocol. • No working Atlas interface — only static mockups and a concept prototype. • No measurements beyond one model, one task, one design factor. • No interaction effect measurements, no white-box measurements, no data on robustness/predictability/safety. • No third-party contributions. • No empirical validation of the taxonomy itself (independent inter-annotator verification remains to be done).
The next 12 months The next 12 months • Infrastructure — finishing the full experimental harness on top of llm-runtime + turnlock for multi-model, multi-task, factorial-design campaigns. • Validation — engaging paid inter-annotators to empirically validate the taxonomy, turning it from a structural proposal into a grounded instrument. • Execution — running Cycle 0 and Cycle 1, populating the registry entry by entry in order of isolability . • Atlas — turning the specification and mockups into a working interactive cartography. • Publication — submitting at least 2 pre-prints for peer review (one for the system design grid, one for the results of the first measurement cycles). • Governance — establishing the qualification standard as a public protocol with versioning, maintenance, and a framework for third-party contributions.
The grid is built, the infrastructure exists, the method works. The next twelve months determine whether the program receives the resources to map the design space systematically.
What we plan to measure first Two measurement cycles are fully specified and ready to execute upon funding. They are designed as a sequence — the second builds on the first — and together they establish both the measurement infrastructure and the first qualified registry entries:
Cycle 0 — Single-call inference surface. One LLM call, one lever moved at a time, no retrieval, no tools, no orchestration . The simplest regime: maximum isolation, maximum signal for the proof-of-method. Cycle 1 — LLM-as-Judge . The system that evaluates another system's output. Both a research object in its own right and the calibration of the scoring instrument that will evaluate every subsequent cycle. Cycle 0 — Single-call inference surface (first) Every measurement runs on a panel of ≥16 models (majority open-weights served by us for full white-box access — seeds honored, logprobs accessible, constrained decoding verifiable) and on two task families (math + structured extraction) to test cross-task generalization.
Core measurements.
ID Design factor Levels Task family Targeted effect
Q-SC Self-consistency (C.2.4 × D.4.4) k ∈ {1, 5, 10, 20} — post-hoc, zero extra generation math + structured (+ satellites, consistency only) effect on task performance exact-match ) + consistency (majority@k stability, flip-rate) Q-FMT Format tax — free vs prompt schema (A.10.2) vs constrained decoding (C.3.2) 3 arms structured (primary) + math (generalization) effect on task performance — correction + format (schema-valid rate = the "tax") Q-DEC Temperature (C.2.1) T ∈ {0, 0.3, 0.7, 1.0} math (primary) + structured + satellites (generalization) effect of temperature on task performance and consistency — the curves Q-THINK Extended thinking on/off + budget (C.4.1, C.4.2) off / on ; budget {low, mid, high} math effect on task performance and consistency — thinking tokens = metadata (cost arbitrage) Q-SEED Determinism (C.2.2) — substrate qualification, not an effect T=0, fixed seed, repeated input math + structured (sample) noise floor
Interaction effects . Beyond main effects, Cycle 0 measures how design factors deform each other — the gradients that make the registry interpolable to configurations not directly measured. Most are post-hoc on the shared pool (zero extra generation):
Interaction Designer question answered Cost
temp × SC At which temperature does self-consistency pay most? post-hoc — free format × SC Does self-consistency recover the format tax? post-hoc — free temp × format Does the format tax depend on temperature? off-diagonal cells (~$25–200) SC × thinking Do self-consistency and thinking stack or substitute? post-hoc on M_reason grid temp × thinking Does thinking ROI depend on temperature? heavy (reasoning tokens) — gated on pilot
What it delivers. Reliability reported as a first-class result (not an error bar), per-item consistency distributions, white-box calibration on logprobs (out of reach of closed-API literature), and interaction gradients making the registry interpolable to configurations not directly measured. Volume: ~1.6–2.5 million generations on open-weights we serve ourselves on rented GPUs (batchable, paid by GPU-hour, not per call) plus ~165,000 thin API calls on closed frontier models (contrast + light anchoring only — the deep replication stays on the open side). Estimated direct cost: ~$10–20k.
Cycle 1 — LLM-as-Judge (second) Why judges, and why now. The field deploys LLM judges everywhere but has skipped a precondition: establishing the judge's own reliability. A judge that doesn't agree with itself has no stable target to validate against a gold standard. Cycle 1 supplies that precondition, measuring each design factor against the judge's own noise floor — on the same panel, same white-box regime, same harness as Cycle 0. Validity is measured via injected defects (oracle B): take a correct answer, inject one known flaw, verify the judge catches it — the only clean, in-regime oracle accessible to a judge.
Core measurements.
ID Design factor Levels Targeted effect
Q-TARE Judge noise floor — substrate qualification, not an effect frozen input, T=0, repeated verdicts noise floor of the judge on open (verified) vs closed (declared)Q-DETECT Injected defect detection (oracle B) clean / defect {conflict, unfounded} × {obvious, subtle} effect on task performance (correctness — detection rate, false positives) Q-POS Position bias (A.7.4 — answer order) direct order / swapped effect on robustness Q-AGG Judge voting (C.2.4 × D.4.4) k ∈ {1, 3, 5, 10} — post-hoc, zero extra generation effect on consistency (majority@k stability) Q-DEC Judge temperature / top-p (C.2.1) low / mid effects on task performance and consistency (curve vs temperature) Q-FMT Format of the evaluated answer (A.7.3) raw / tagged / reformatted effect on robustness (invariance to reformatting) Q-VERB Verbosity — padding of the evaluated answer concise / padded effect on robustness (does the verdict follow length?) Q-RUBRIC Rubric granularity (A.4.4) coarse / fine effects on consistency and task performance Q-COT Judge reasoning — native (C.4.1) or prompted (A.5) off / on (budget) effects on task performance and consistency — reasoning tokens = metadata Q-SELF Self-preference (C.1.1 — does the judge favor its own lineage?) judge's lineage vs other lineages (band) excess score granted to own lineage — secondary
Interaction effects.
Interaction Designer question answered Cost
temp × vote-k Does the value of judge voting depend on temperature? post-hoc — free difficulty × vote Does voting recover subtle defects more than obvious ones? post-hoc — free position × verbosity Does position bias depend on how verbose the answer is? off-diagonal cells (moderate) format × temp Does format sensitivity depend on temperature? off-diagonal cells (moderate) CoT × difficulty Does reasoning help mostly on subtle defects? heavy (reasoning tokens) — gated on pilot
What it delivers. The first reliability-qualified judge substrate, a catalog of biases reformulated as measured effects (with confidence intervals, against a noise floor, across ≥16 judges), and the scoring instrument that Cycle 2 (RAG ) inherits. Volume: ~3.2–3.8 million verdicts on open-weights plus ~145,000 thin API calls on closed judges — the closed side is even lighter than Cycle 0 because closed models are structurally excluded from white-box dimensions (no logprobs, no verified seed) and self-preference (no lineage in the generator pool). Estimated direct cost: ~$7–16k.
Why not agentic AI yet Agentic AI co-activates five of the six system-design families at once — retrieval, tools, orchestration, output processing, memory — making it impossible to isolate any single design factor . Change one thing and everything shifts; the result is a score, not an attributed effect. The program therefore builds the substrate in ascending order of co-activation : single-call (Cycle 0, A+C only) → LLM-as-judge (Cycle 1, +D for scoring) → RAG (Cycle 2, +B for retrieval). Agentic regimes enter once the simpler substrates are populated — because measuring an agent without a noise floor, a calibrated scoring instrument, and a characterized retrieval baseline would produce exactly the confounded results Rhondda exists to replace. Each cycle defers its successor explicitly; it's not procrastination, it's the only honest path to composable measurements.
Why this sequence Cycle 0 pays the fixed costs — experimental harness, measurement doctrine, noise floor — on the cheapest possible regime. Cycle 1 extends that same harness to the judge for a fraction of the cost. Cycle 2 (RAG) inherits both the noise floor and a qualified scoring instrument. Each step carries the previous one; skipping ahead means measuring complex regimes with uncalibrated tools.
Both cycles are designed to be solvable solo — the compute is batchable and the engineering bottleneck is model wrangling, not team size — and compatible with the scope of an Emergent Ventures grant.
How to make it a standard Rhondda/Atlas has no value unless adopted. It aims to become a public substrate where field knowledge accumulates, and that only exists if many contribute to it and use it. Kept to oneself, it would remain just another local tool. Aiming for mass adoption is therefore not a secondary ambition: it is the project's very condition of existence.
Rather than leaving this to chance, the aim is to understand how to be proactive. The approach of this document is thus to understand the mechanisms that govern mass adoption of a tool like Rhondda/Atlas, in order to derive a proactive strategy.
There are two historical precedents Rhondda/Atlas can draw on, because both achieved the very goal we're targeting: becoming the massively adopted standard in their field. Those precedents are MITRE ATT&CK for cybersecurity, and evidence-based medicine (EBM) for clinical medicine. This is exactly what Rhondda/Atlas aims to become for AI system design. We first lay out the facts, then draw the lessons afterward.
Lessons from Cochrane and MITRE ATT&CK MITRE ATT&CK: From Measurement Instrument to Universal Language
In 2013, MITRE set out to build a measurement instrument: a way to assess how well an organization detects known adversary behaviors. The project, called the Fort Meade Experiment, had a practical goal: evaluate detection coverage. But the instrument immediately revealed a deeper problem. The offensive team emulating attacks and the defensive team trying to detect them had no shared language. What one produced, the other couldn't map to what it was looking for.
The solution was a coordination language: a taxonomy of adversary techniques. Stabilizing that language produced the ATT&CK matrix, published in 2015 with 96 techniques across 9 tactics. It was full and consultable on day one.
Then came the adoption pattern that matters most. MITRE published the matrix and stepped back. No top-down mandate, no prescribed usage. Adoption was slow, near silence at first, and entirely bottom-up. Security operations centers used it to map their detection coverage and spot their gaps. Threat intelligence teams tagged their reports with ATT&CK techniques. Red teams structured their emulations around it. Each organization extracted value solo, without needing anyone else to adopt first.
The field-wide common language came later, as an effect of diffusion, not its cause. What turned widespread use into a de facto standard was a mechanism that arrived around 2018, three years after publication: ATT&CK evaluations. When vendor reputations began transiting through these coordinates, and RFPs started demanding ATT&CK coverage, not speaking the language became costly. That cost of abstention is what made ATT&CK massive, not just popular.
Evidence-Based Medicine: The Grievance, the Method, the Institution
EBM began with a grievance, not a tool. In 1972, epidemiologist Archie Cochrane identified a paradox: medicine produced clinical trials, but no one systematically gathered, graded, and synthesized them. The knowledge existed; it just didn't accumulate. Cochrane didn't create the need; he named it.
From that grievance came two things, in sequence. First, a teachable method: at McMaster University, David Sackett and colleagues transformed the grievance into a learnable practice: how to read, grade, and apply evidence. The term "evidence-based medicine" was coined around 1991. Second, a durable institution: Iain Chalmers founded the Cochrane Centre in 1992 and the Cochrane Collaboration in 1993, whose entire function is to synthesize, version, and update evidence. This is something ATT&CK never had to build: an attack technique stays stable; a medical finding requires revision.
Adoption took decades, and no single actor drove it. The binding mechanism, the equivalent of ATT&CK evaluations, came from outside: integration into clinical guidelines and reimbursement policies. Once insurers and health systems tied payment to evidence, deviation became costly. That lever arrived long after Cochrane's grievance, just as ATT&CK evaluations arrived long after the matrix.
The Pattern: Measurement Instrument → Cartography → Language
Both trajectories reveal the same triptych:
Measurement instrument : what produces the data. For ATT&CK, the detection evaluation from Fort Meade. For EBM, the clinical trial and the grading method.Cartography : the published substrate where the field locates itself. The ATT&CK matrix; the Cochrane corpus.Coordination language : what the cartography becomes once the field adopts it. A shared vocabulary for cyber defense; graded evidence as the common reference for clinical decisions.Transposed to Rhondda/Atlas: Rhondda is the measurement instrument, the machine that produces attributed, reproducible effects. Atlas is the cartography, the published substrate consulted solo to locate oneself and spot gaps. The coordination language is not a separate object; it's the function Atlas fills once adopted.
Six Lessons Both Precedents Agree On
1. The need pre-existed; the tool formulated and served it, not created it. Describing attacks in a common language was already necessary across organizations; cumulating medical evidence was too. You don't push an object ahead of the need that anchors it.
2. Authority shifts from the person to the evidence. An entry is trusted for what it rests on: actually observed attacks, graded evidence quality, not who published it. This is the exact displacement Rhondda/Atlas targets: from "my opinion" to canonical, empirically anchored data.
3. The trajectory has an order, and you don't skip steps. Internal genesis → public substrate → adoption → binding mechanism. ATT&CK: 2013 → 2015 → adoption → evaluations 2018. EBM: grievance 1972 → method and institution → diffusion → reimbursement.
4. What massifies adoption is an external mechanism that makes non-use costly, and it arrives late. Evaluations and RFPs for ATT&CK; guidelines and reimbursement for EBM. In both cases, this mechanism is posterior to adoption, not its entry point.
5. Adoption is not decreed; it is slow, decentralized, and appropriated by its users. Years for ATT&CK, decades for EBM. No single actor steered it. The carrier produces and publishes; they make adoption possible, they don't control it. A slow start is not failure.
6. At bottom, the same dyad: the one who measures ↔ the one who reuses. The researcher who deposits a trial and the clinician who applies it without redoing it. The analyst who documents a technique and the one who consumes it elsewhere. This is the dyad Rhondda/Atlas must coordinate.
What Each Precedent Adds
From ATT&CK:
A full map is the precondition. ATT&CK was published full: 96 techniques, 9 tactics, consultable on day one. Rhondda/Atlas starts empty (zero validated effects). The measurement cycles aren't a side activity; they are what makes adoption possible. No filled map, no ignition.Solo value ignites the first wave. Mapping your own coverage, spotting your own gaps, alone and without anyone else adopting, is what lit the first wave. MITRE built ATT&CK to coordinate two teams; the field adopted it for entirely different reasons. The adoption motive is not the creation motive.From EBM:
The conditional, graded regime: exactly Rhondda/Atlas's regime. A piece of evidence is dated, graded, reported under conditions, and subject to revision. This is precisely the status of Rhondda/Atlas effects, verifiable but perishable, that ATT&CK's stable units cannot illustrate. EBM made it respectable to say "this is only directional in your context": the sign and structure of the effect, not its exact value.The cumulative corpus held by a dedicated institution. Cochrane synthesizes, versions, updates: a profession of synthesis, not a one-time publication. This is the institutional brick, "Rhondda/Atlas's Chalmers," that must eventually be named.The inversion of the burden. A practice is justified by evidence, rather than maintained until proven harmful.Where Rhondda/Atlas Stands Today
Rhondda/Atlas is at the exact starting point both precedents once knew: an object not yet adopted. The question is not commercial. It's a matter of diffusion and, upstream, of locating where the need is already felt.
That question splits in two. How to ignite adoption : the solo value each organization extracts, which triggers the first wave — security teams mapping their coverage against ATT&CK, clinicians consulting Cochrane alone. How to massify it : the network effect that composes solo users into a de facto standard — the moment not speaking the language becomes costly.
The order is not negotiable. Instrument and cartography come first; "language" only enters once we know who adopts and for what. Confusing which role matters at which moment is what breaks the argument. The two precedents didn't skip steps, and neither can Rhondda/Atlas.
How to ignite adoption Proactive, Not Passive
The two precedents diverged on one critical choice. MITRE published ATT&CK in 2015 and waited: adoption emerged on its own, slowly, over years. Its active community-building came later, accompanying massification, not triggering ignition. EBM did the opposite. At McMaster, its carriers actively brought the method to practitioners through courses, residency programs, summer workshops, and a published series of methodological guides. They taught and published their way into adoption; they didn't hope for it.
I follow EBM. I don't have a university behind me, but I have the same levers: publish, demonstrate, contact directly. The goal is not to wait and see who picks it up. It's to locate the designer for whom "good enough" has stopped being enough, and show them what the substrate already knows about their problem.
What the Precedents Tell Us About the Early Adopter
Both precedents converge on the same profile, and it's not abstract. The early adopter is not a manager or a buyer. It's a practitioner — someone whose job already involves measuring or evaluating, operating in a high-stakes environment , for whom the substrate resonates with a lived pain . For ATT&CK: SOC analysts and detection engineers, people whose daily work was mapping detection coverage. For EBM: clinicians with methodological training at McMaster, confronting visibly inconsistent practice. In both cases, the adopter was already doing the work; the substrate gave them a better way to do it.
The stakes themselves come from two distinct engines. The ATT&CK way : exogenous and consequential. A real adversary exploits gaps; failure costs in breaches, ransomware, regulation. The threat imposes the stakes; no one needs convincing. The EBM way : self-imposed and cultural. A community holds itself to the standard of evidence before any regulator demands it, because the inconsistency of unverified practice is itself the stakes. The strongest ignition point sits at the intersection : a product where measurement has real consequences, carried by a team that takes measurement seriously.
EBM also left a subtler lesson. It didn't just serve a pre-existing stakes; it made the stakes visible . By showing that unverified practices were inconsistent, it turned a latent problem into a recognized one. Part of the proactive work for Rhondda is the same: demonstrate that evaluation decisions are being made on noise, and that an attributed measurement changes what's knowable.
Why the First Cycle Targets the LLM-as-Judge
We chose the LLM-as-judge as the first measurement cycle for methodological and technical reasons first, detailed in What we plan to measure first . But it is also the field where latent demand for what Rhondda produces is strongest and most explicitly expressed. I've spent months deep-diving the academic and technical literature on LLM-judges. What the field calls the evaluation crisis is real and documented: anyone deploying an LLM-judge questions its reliability , its bias, its stability across runs, and has no proper answer. The cost of uncertainty is already being borne in missed regressions, phantom gains, and calibration loops that never converge. An identifiable population is already paying for the very thing the first cycle measures. This is not a coincidence. Because the judge will serve as the scoring instrument for most of Rhondda's subsequent cycles, measuring it first calibrates the tool before it's used everywhere else. Methodological necessity and field-wide pain point to the same starting line. That overlap is what makes ignition possible.
Who Operates LLM-Judges Today
The archetype from the precedents is a portrait, not a map. To find the real target, I need to map who actually operates LLM-judges in production today. Five families:
Product teams build LLM applications and deploy LLM-judges in CI/CD to gate deployments. The judge is a means to an end: an internal instrument that says whether a change passes or breaks. The largest group, and the most dispersed. Eval platforms — Braintrust, LangSmith, Arize Phoenix, Langfuse — provide evaluation capabilities, including ready-made LLM-judges, to product teams. The judge is the core of their offering; its reliability is their commercial value.Model providers use judges internally for evaluation and RLHF/RLAIF training. Near-unlimited resources, low dependence on any external substrate.Research and meta-eval — JudgeBench, RewardBench — study the judge itself as an object. Strong methodological culture, built to publish and share.Regulated actors — health, finance, legal — deploy LLM systems where a wrong answer costs heavily, and use judges to control and document outputs. Maximum stakes, but long adoption cycles make them poor first adopters for a young substrate.All five share the same two-step practice. They choose their judge configuration using known best practices — randomizing order, majority voting , few-shot examples, binary scoring — and they validate by calibrating against a human-annotated golden set. None starts from scratch. But none of this is a stable foundation. These best practices are directional regularities glimpsed in scattered work or team experience, never isolated under a noise floor , without declared application conditions, perishable under a new model. Calibration validates a config; it doesn't tell you which lever to pull. Choosing the configuration itself, without Atlas, means sweeping all combinations (infeasible) or guessing (unquantified). The very deficit the substrate exists to fill is what teams currently build on.
Who to Target First
I filtered the five families through seven criteria: intensity of the direct gain, measurement autonomy, leverage effect, value to the substrate, competition risk, accessibility, and above all whether "good enough" has genuinely stopped being enough. Here is how they sort:
Model providers : excluded. Near-unlimited resources to remeasure everything themselves, hard to reach as a solo actor, no incentive to contribute proprietary advantage.Product teams with real stakes and room to experiment : the ignition segment. They feel the direct pain, lack the resources to rebuild the prior themselves, have no competitive reason to hoard, and are numerous enough to be reachable.Research and meta-eval : engaged early, but as contributors who grow the substrate — the way trialists fed Cochrane — not as consumers to ignite.Eval platforms : reserved for massification . Their leverage is enormous (a reliable judge diffuses to all their clients) but they have both the means to remeasure everything and the temptation to build proprietary meta-eval, making a public, neutral substrate the right angle of approach. In one sentence: ignite with product teams who use LLM-as-judges under real stakes, engage research as contributors, reserve platforms for massification, exclude model providers.
The Crystallized Target
The six constraints above — the posture, the archetype, the stakes, the scope of the first cycle, the families, the sorting — don't point to a single job title. The person might be an eval engineer, an MLE, an applied scientist, a platform engineer, or a tech lead who inherited the pipeline. What defines them is the situation.
They built an LLM-judge to continuously score their system — an assistant, an agent, a RAG pipeline — at every change. They already did the sensible things: randomized candidate order, set up majority voting, calibrated against a few dozen human-annotated examples. One day they tweak a setting, watch the score move by two tenths, and realize they don't know whether their change paid off or the judge simply wobbled. Everything they've done has brought them to this point and abandoned them there.
What they need is not another round of calibration. It's knowing whether, under conditions close to theirs, a specific lever — voting at k, switching to binary scoring, fixing the output format — reduces variance, and by how much. That's what they come to Atlas for: not to skip calibrating on their own data, but to stop re-testing blind what the field has already established.
What distinguishes them from the multitude isn't the wish to measure better. Nearly every team has that, and nearly every team calibrates, reruns, adjusts by feel, and moves on. What distinguishes them is that a deployment decision costly enough makes "good enough" no longer tenable. The pain is active , not latent.
That's the bet.
The Signal
The signal that ignition has happened is not page views or sign-ups. It's unitary and concrete: someone made a decision they wouldn't have made otherwise, based on a measurement they didn't have to redo , because the field had already established it in terms they could reuse. The day that happens, the need for Atlas moves from thesis to demonstrated use case. Until then, whatever the traffic numbers say, ignition hasn't happened.
How to massify adoption We're Not There Yet
Mass adoption is the project's condition of existence, but right now we're still at the previous stage: igniting the first adopters. What follows is therefore not a detailed plan. It's the strategic principle derived from the two precedents, condensed — the logic we'll follow once ignition has taken hold.
The Core Principle
Both ATT&CK and EBM teach the same lesson: what turns isolated users into a de facto standard is an external mechanism that makes not speaking the language costly. For ATT&CK, it was evaluations and RFPs demanding coverage. For EBM, it was the integration of graded evidence into clinical guidelines and, decisively, into reimbursement: once insurers and health systems tied payment to evidence, deviation became costly. In both cases, this mechanism arrived late — years after the substrate was already adopted — and it was not controlled by the substrate's creator.
The strategic question for massification is therefore: what will make not using Atlas coordinates costly? Two routes exist. Either find an existing situation where this cost already exists and plug into it — fast. Or fabricate it — slow, and risky for a solo actor. The bet is to find the first or fabricate the second at the lowest possible cost.
Three Mechanisms, Three Scales
The analysis of both precedents points to three candidates that can carry this cost, at three increasing scales:
The internal demander (one org). A technical lead or CTO who imposes a standard of proof that references the substrate: "I won't accept your claim unless you show it against what Atlas knows." This is the smallest fabricable unit — a single person can impose it unilaterally — and the most plausible germ of contribution, because the same lead who demands internal reuse ("team B must not re-measure what team A established") seeds an internal substrate that can later federate into Atlas.
The third-party evaluator (the market). Eval platforms already sit across hundreds of systems. If designers demand evaluation in Atlas coordinates at purchase, vendors have a commercial interest to report in the language — exactly the ATT&CK mechanism of RFPs demanding coverage. The trap: reporting in the language is not the same as depositing into the public substrate. The platform's accumulated data is its moat; a public Atlas competes with it. The flywheel has a fragile link, and bridging it requires either an open-core model (moat moves to the decision-support layer), a challenger strategy, or late table-stakes pressure.
The metrology institution (the field). EBM wasn't built by one person. Cochrane named the grievance, Sackett made it teachable, and Chalmers built the Cochrane Collaboration — a durable institution whose profession is to synthesize, version, and update evidence. Today I carry the first two roles. The third — Rhondda's Chalmers — is the missing brick: a reference institution that deposits attributed effects into the public substrate by construction. The cycles Rhondda is already running are its germ.
The Preferred Path
Creating the institution at full scale from day one is beyond a solo actor. Converting an existing platform is faster but depends on finding a volunteer. The preferred path is to germinate first, convert later . Creating the institution at full scale from day one is beyond a solo actor like me. Converting an existing platform is faster but depends on finding a volunteer. So: germinate a thin metrology institution — the cycles plus their synthesis — to establish the standard's credibility, then convert incumbents once deposit becomes table-stakes. The reference evaluator already exists in germ. The real question isn't whether to create it; it's how to grow it from one person running cycles into an institution others feed and trust.
How will I find my early adopters? I plan to publish pre-prints and articles around the measurements as they come, and build an audience from that content. Without a published result, there's nothing to show when I knock on a door. The pre-prints are the entry ticket for every conversation that follows.
In parallel, I run a prospecting effort to find and recruit the first design partners — the people who will give early feedback on Atlas as a navigation tool, not contribute content — using two concentric circles ordered by proximity to the pain: the noise, bias, and instability of LLM-judges that makes evaluation decisions unreliable. The plan covers what I say, where I look, how I track leads, and in what order I move. Much of the extraction and initial filtering can be automated.
The Pitch
When I contact someone, I don't pitch Rhondda. I describe what I'm building and what I'm looking for. The pre-print is the entry ticket — not an argument, a result. What follows is a sketch; the pitch still needs to be refined and tested, but this is the idea.
"I'm leading Rhondda, a measurement program that isolates what happens when you change one thing in an AI system's design — starting with the LLM-as-judge . I publish pre-prints of the results as they come. Atlas is the interface that will make those results navigable: pick a lever, pick a model, see what the data says and under which conditions it holds. I'm looking for design partners for the beta of Atlas — people who already run an eval pipeline and know what they'd want to find in a tool like this. Just feedback on the navigation, so it answers real questions. Does this speak to you?"
I chose this approach for three reasons:
It gives me a genuine pretext for the conversation. People are far more willing to talk when they feel valued ("your opinion matters, help me build this for you") than when they're pitched. If the conversation goes well and they sign up for the beta, I'm building a contact list, an audience, and a concrete signal of interest for Atlas — something I can leverage to raise more funds, recruit, and open doors. It's honest. I genuinely need their feedback to build Atlas right. The Two Circles
I found these circles by analyzing the landscape. This is where I'll prospect — each circle intersects the target archetype at a different distance from the pain. I chose them because they are visible, reachable audiences : public communities, active Discords and Slacks, people who post and write under their own names. I don't need an introduction or a gatekeeper. The list is meant to be explored in depth, not to be exhaustive.
Circle 1 — Measurement culture (entry point). The public that already imposes rigor on itself and knows how to read an attributed measurement.
Who Context Hamel Husain & Shreya Shankar Creators of the most-followed evals course in the field (4,500+ students from OpenAI, Google, Meta, Anthropic). Their free event The Cold-Start Eval Problem on July 16, 2026 is the entry door. Eugene Yan Senior Applied Scientist at Amazon. Reference blog on LLM evaluation. Author of "Stop Blaming the LLM-as-Judge; Fix Your Process Instead." Simon Willison Creator of Datasette, reference figure in LLM tools and evals. Highly active, maximum visibility.
Angle: the pitch described above. I share the pre-print, I ask for feedback on Atlas as a navigation tool. This public knows how to read a measurement; the value demonstrates itself.
Circle 2 — Operational pain (second). The communities where engineers tune their judges in real time and post "how do I reduce my variance," "which scoring format," "position bias."
Platform Context Braintrust Eval-driven development with CI/CD gates — the exact profile of the target archetype. Active Discord. Arize Phoenix Open-source, eval-oriented. Active Slack. LangChain / LangSmith The LangChain ecosystem. Active Discord. Langfuse Open-source (MIT), YC W23, ~4,100 members. Active Discord.
Angle: same pitch, but only once the judge cycle is published and I have concrete results to show. I target the user, never the platform. I enter through their specific tool question — answering a help thread, not pitching cold.
Separately, Reddit, Hacker News, and X are not approach channels. They're a radar — where the pain surfaces unfiltered, and where I spot when and where to intervene.
Why me 1. I came to this problem by building real systems that hit a wall: changing one parameter and having no way to predict the effect. That means I'm not solving a problem I read about. I'm solving one I've been bleeding on for a year.
2. Rhondda is my first verifiable track record and I don't have a title to hide behind. So, every decision has to hold up to scrutiny because my work is the only credential I have. That's a built-in quality check.
3. This grant isn't a condition, I'm building either way. But without it, the field waits three years for a standard it could have in twelve months.
4. I have no academic career, no model-provider ties, no lab expecting papers. The questions I ask don't have to be career-safe. The answers I build don't have to please a reviewer or a partner.
5. The field rewards novelty; nobody rewards maintenance. I don't need to be rewarded, I need answers & proofs. If it means versioning a registry, re-qualifying metrics when models drift, and publishing release notes, I'll do it. I'll do it for years.
6. I'm designing Rhondda to outlive me: governance, contribution protocol, licensing. Which means the grant funds infrastructure that won't depend on the grantee. My plans are not to be the bottleneck forever.
7. I have a hypothesis I need to test ASAP about who my first users are and how to reach them. I treat adoption at the same level as the technical work.
8. I need to know how far I can go when I put everything I have behind a problem I chose.
