A dashboard can show a higher conversion rate without proving that a variant caused the difference. The result still has to survive checks for assignment, tracking, scope, sample quality, implementation changes, and reporting language.
Use this checklist before an experiment result becomes a client update, sales deck, executive summary, roadmap decision, or public claim. It is designed to identify what is supported, what needs a caveat, and what should not be reported yet.
1. Write the exact claim first
Start with the sentence the team wants to send. A specific claim exposes the evidence it requires.
The first sentence requires evidence about variant exposure, the definition of a completed demo request, the date range, and whether the percentage describes absolute or relative lift. If those details are unavailable, the wording has already outrun the evidence.
2. Check assignment and exposure
Confirm that each included visitor could actually see the variant assigned to them. Then compare observed traffic with the intended allocation. A material split imbalance can point to redirect failures, audience rules, campaign routing, browser exclusions, bot filtering, or missing assignment events.
- The experiment id and variant id are present in the evidence.
- Both variants received traffic throughout the stated test window.
- Observed exposure is reasonably consistent with the intended split.
- A visitor is not counted in more than one variant unless the design explicitly allows it.
- Internal, QA, bot, and preview traffic are understood or excluded consistently.
- Campaign and landing-page rules did not send materially different audiences to each variant.
3. Verify conversion linkage
The primary conversion must mean the same thing for every variant and be linked to a known assignment. A click, form attempt, successful submission, qualified lead, checkout start, purchase, and retained customer are different events.
- The primary metric has one written definition.
- The same event or outcome is used for every variant.
- The event fires after variant assignment is known.
- The event fires once for the intended action, not on validation errors or repeat renders.
- Backend-confirmed outcomes are distinguished from browser-side attempts where relevant.
- Event names, parameters, destinations, and trigger rules remained stable during the test.
4. Reconcile scope and denominators
Many apparent disagreements are denominator disagreements. One system may report users, another sessions, another page views, and another clicks. A rate cannot be interpreted until its numerator and denominator are explicit.
| Question | What to record |
|---|---|
| Who or what is counted? | Users, sessions, exposures, clicks, orders, leads, or events. |
| Which population is included? | Date range, country, source, campaign, page, device, consent state, and exclusions. |
| What is the numerator? | The exact event or business outcome counted as a conversion. |
| What is the denominator? | The exact eligible population used to calculate the rate. |
| How is attribution handled? | Session, user, click, view-through, or platform-specific attribution. |
5. Check for implementation drift
A clean aggregate can hide a broken first week or a tracking change halfway through the run. Review the release and configuration timeline alongside the result.
- Experiment code, landing pages, forms, checkout, or campaign routing did not materially change unnoticed.
- GTM container versions and GA4 event definitions are known for the full window.
- Consent behavior did not change the measured population differently by variant.
- Outages, partial deployments, or data-import delays are recorded.
- If implementation changed, the report uses a clean post-change window or states the limitation.
6. Match the wording to the evidence
The final output does not have to be winner or loser. A useful evidence review can return one of several honest outcomes.
| Outcome | Appropriate wording |
|---|---|
| Supported | The supplied evidence supports the stated result within the recorded scope. |
| Directional | The observed result is promising, but the evidence does not support a confident winner claim yet. |
| Report with caveats | The result may be shared only with the stated tracking, scope, or sample limitations. |
| Inconclusive | The supplied evidence does not distinguish the variants confidently. |
| Tracking invalid | The result should not be reported until the measurement problem is resolved. |
The minimum evidence pack
For an independent review, package the records that let another person reconstruct the claim without relying on the dashboard headline.
- The proposed report sentence and primary decision being made.
- Experiment name, variants, allocation, dates, and audience rules.
- Exposure and conversion counts by variant.
- Metric definitions and calculation method.
- Relevant GA4 explorations or exports and GTM version context.
- Testing-tool export and any CRM, ecommerce, or ad-platform comparison.
- Known implementation changes, exclusions, and caveats.