The experiment dashboard says the new checkout converts better. It also says fewer visitors saw it, even though allocation was configured as 50/50. Before discussing the uplift, reconcile the population.
Sample ratio mismatch, or SRM, asks whether the observed split is unexpectedly far from the intended allocation. It does not mean every unequal count is a fault. Random assignment produces unequal counts naturally. Nor does it identify the cause of an unusual split.
The practical task is to find where people disappear, appear twice or change eligibility between assignment and the final report. That work requires several counts, not another look at the conversion-rate chart.
Start with the population the test promised
Write down the randomization unit, eligibility rule, allocation ratio and relevant dates. A test assigning people should be reconciled using distinct people, not pageviews. A configured 90/10 rollout should be compared with 90/10 expectations. A changing allocation needs a time-aware analysis rather than an assumed full-period 50/50 split.
Microsoft’s SRM guidance describes problems arising across assignment, experiment execution, data processing and analysis. It uses a goodness-of-fit test against the intended ratio, then diagnostic segmentation to investigate the signal. The size of the sample matters alongside the observed percentage split.
Keep a record of configuration changes. If someone altered eligibility midway through the experiment, the dashboard’s present settings may not explain the earlier traffic. Establish what actually ran before choosing the expected counts.
Follow the same people through four stages
Here is a synthetic example with a deliberately injected logging fault. These counts are illustrative, not results from a real checkout.
| Stage | Control | Treatment | What the count represents |
|---|---|---|---|
| Assigned | 5,000 | 5,000 | Distinct eligible people assigned to an arm |
| Exposed | 4,800 | 4,800 | People reaching the designated exposure point |
| Logged | 4,800 | 4,400 | Exposure records arriving in the analysis pipeline |
| Analyzed | 4,800 | 4,400 | People retained by the final report query |
The first unequal step is between exposure and logged records. In this fixture, 400 treatment exposure records are lost. The report then reproduces that loss faithfully.
That locates an investigation; it does not prove that every real discrepancy at this boundary is logging loss. The example supplies exposure counts independently so the gap can be seen. If “exposed” and “logged” come from the same broken event, their apparent agreement would tell us little.
Ask where each count originates. Assignment might come from the experiment service. Exposure might come from a separate server record. The analysis population might come from a warehouse query. Preserve stable identifiers so those stages can be reconciled, subject to the organization’s data-handling rules.
Calculate the alert without treating it as a diagnosis
For the final 9,200 recorded people, a 50/50 expectation gives 4,600 per arm. The Pearson chi-square statistic is:
(4,800 − 4,600)² / 4,600 + (4,400 − 4,600)² / 4,600 = 17.3913
With two categories and a specified allocation ratio, that gives one degree of freedom and a p-value of approximately 0.0000304. The NIST goodness-of-fit reference provides the observed-versus-expected formula and notes that the approximation requires sufficient sample size.
Suppose this team chose an alert threshold of 0.001 before examining the result. The fixture would trigger it. That threshold is an illustrative operating policy, not a universal definition of acceptable experimentation.
The p-value is not the probability that logging is broken. It describes how surprising a discrepancy at least this large would be under the specified allocation model and test assumptions. It cannot tell you whether the cause is assignment, missing events, filtering or a mistaken expectation.
Segment to localize the missing records
The fictional logged population can be split by device:
| Device | Control | Treatment |
|---|---|---|
| Desktop | 2,400 | 2,400 |
| Mobile | 2,400 | 2,000 |
The entire gap sits in mobile. That gives the engineering investigation a concrete starting point: compare the mobile exposure-to-ingestion path between arms. Check the relevant release, event payload, request failure and processing rules.
Do not turn this into a search for a segment where the treatment wins. Segmentation here is diagnostic. It helps identify which records need inspection and which code path could explain their absence.
Also distinguish a property established before treatment from behavior that treatment can change. Device at assignment may be a useful diagnostic dimension. “People who reached payment” can be affected by the checkout itself. An uneven split among downstream finishers may reflect real behavior; it is not automatically evidence of bad randomization.
That is why the exposure definition matters. If treatment changes whether someone qualifies for analysis, selecting only those who qualify can change the comparison population.
Fix the cause before choosing a recovery method
In the synthetic case, the repair is to restore the lost mobile records or correct the faulty logging path. Dropping 400 control visitors to make the columns equal would conceal the symptom while leaving the selection problem unexplained.
Do not assume a rerun is always necessary or that backfilling is always safe. Recovery depends on what remains available. If complete source records exist, the team may be able to rebuild the analysis using a documented correction. If the missing records cannot be recovered reliably, a clean rerun may be the defensible choice.
Record which interval, devices and metrics are affected. A fault confined to one event can have a different impact from missing entire users. Check whether order outcomes still exist independently and whether they can be joined to the original assignment without selecting only successful purchasers.
Close the investigation with a reconciled readout
The next report should explain the expected allocation, the first divergent stage, the diagnosed cause, the repair and the remaining limits. Include counts before and after the correction so another person can understand what changed.
Then revisit conversion performance using the agreed analysis population and observation window. An SRM investigation may reveal that the apparent winner was benefiting from missing nonbuyers. It may also show an incorrect alert expectation while the underlying experiment remains usable. Neither conclusion should be chosen in advance.
A tidy 50/50 column is not the objective. The objective is a comparison whose population can be explained from assignment through outcome, with the missing-record question settled before the commercial decision is made.