A visitor assigned yesterday has had less time to buy than a visitor assigned last week. If an experiment report counts both as fully observed nonbuyers, its conversion rate mixes people at different stages of the buying decision.
This becomes especially important when a checkout change affects purchase timing. One version may encourage an immediate order; another may lead people to return later. A one-day outcome and a seven-day outcome can tell different, legitimate stories.
Choose the question first: purchase within how long after which starting event? Then give the people in that comparison the same opportunity to produce the outcome.
Define a forward-looking observation window
For a visitor-level experiment, a workable definition might be: “The proportion of assigned eligible people who complete their first qualifying purchase within seven days of initial assignment.”
That sentence specifies the unit, outcome, starting point and horizon. It still needs operational details: the purchase status that qualifies, identity handling for return visits, timezone conventions and whether the boundary includes an event exactly at the end of the window.
In the worked example below, assignment is the starting point. A purchase qualifies when it occurs at or after assignment and before the horizon ends. A visitor enters the seven-day readout only after seven full days have elapsed. Each person counts once.
Microsoft’s pre-experiment guidance emphasizes specifying the experiment’s hypothesis and metrics before launch. The observation window belongs in that specification because changing it can change which behavior the result measures.
A timestamped example: the tie that changes meaning
Consider 20 fictional visitors, split equally between A and B. Five people per arm are assigned at midnight UTC on January 1; another five per arm are assigned at midnight UTC on January 7. The dates and outcomes are illustrative.
For each five-person cohort, the stipulated first-purchase pattern is:
| Arm | Purchase timing after assignment | People who do not purchase |
|---|---|---|
| A | Two people after 12 hours; one after 4 days | 2 |
| B | One person after 12 hours; three after 3 days | 1 |
Now read the experiment at midnight on January 8. The January 1 cohort has seven days of follow-up. The January 7 cohort has one day.
An unrestricted “purchased by now” count gives five purchasers out of ten assigned people in each arm. A appears to tie B at 50%. But the denominator combines two different observation ages, and the slower purchases from the later cohort have not happened yet.
Two fixed-window readouts at that same cutoff tell us more:
| Readout on January 8 | Eligible people per arm | A purchasers | B purchasers |
|---|---|---|---|
| Full one-day window | 10 | 4 of 10, or 40% | 2 of 10, or 20% |
| Full seven-day window | 5 | 3 of 5, or 60% | 4 of 5, or 80% |
The seven-day row includes only the January 1 cohort. It does not silently count the younger cohort as nonconverting. On January 14, the later cohort also has seven full days of follow-up, producing six purchasers out of ten for A and eight out of ten for B.
These small, constructed counts show a timing issue. They do not establish a reliable treatment effect or supply a sample-size recommendation.
Do not use tomorrow’s purchase to improve today’s readout
A historical analysis can accidentally join every later order to an earlier assignment, even when the claimed reporting cutoff came before the order occurred. That would give the analyst information the report could not have contained at the time.
Apply both boundaries: the purchase must fall within the visitor’s outcome window and must have occurred by the reporting cutoff. Eligibility also depends on whether the full window has elapsed, unless the chosen method explicitly handles incomplete follow-up.
For a simple mature-cohort analysis, the logic is:
eligible = assignment_time + horizon <= report_cutoff
converted = assignment_time <= first_purchase_time < assignment_time + horizon
Also require first_purchase_time <= report_cutoff. With fully mature cohorts that last condition should follow naturally, but retaining the check makes the intended historical boundary explicit.
Keep the data cutoff separate from the time the query is run. A report regenerated later should not quietly acquire purchases that occurred after its stated cutoff.
Attribution lookback answers another question
Analytics products also use the phrase “conversion window” for attribution. Google’s key-event lookback documentation describes how far backward from an event a touchpoint can remain eligible for attribution credit.
That is a backward-looking credit rule. The experiment definition above follows a person forward from assignment to determine whether a purchase occurs within a fixed horizon. Adjusting an attribution setting does not, by itself, make an experiment’s visitors equally mature.
Write both definitions where the team can see them. For example, marketing may use one attribution lookback while the product experiment measures first purchase within seven days. They can coexist, provided a report does not present one as if it were the other.
Choose the horizon for the decision
Use the buying process and available historical purchase timing to inform the horizon before the result is visible. A longer window can include more delayed purchases, but it also requires more waiting for recently assigned people to mature. A shorter one answers a faster-purchase question and may miss later outcomes that matter commercially.
If several horizons are useful, designate the primary decision window and treat the others according to the planned analysis. Do not select the most favorable window after seeing that one day favors A and seven days favors B.
Preserve original assignment for returning visitors under the experiment’s identity rules. Reassigning someone on each visit can detach a later purchase from the experience that began the comparison. If cross-device identity is incomplete, state which purchases the analysis can and cannot connect.
Report maturity alongside conversion
A clear readout names the enrollment period, report cutoff, horizon and number of mature people in each arm. Show how many assigned people are still waiting for their full window, rather than mixing them into the final rate without explanation.
Equal follow-up solves a specific timing problem. It does not guarantee comparable populations after other exclusions, sufficient statistical power or a stable treatment effect across calendar periods. Check those issues separately before making a release decision.
Finally, keep later refunds outside this initial-purchase question unless the metric explicitly incorporates them. A person can convert within seven days and receive a refund later. First-purchase timing and retained revenue are both useful outcomes, but each deserves its own clock and definition.