All A/B tests were turned off Wednesday June 11 at ~7pm PT. This report tracks hourly conversion rate before and after — sitewide and on the rise-2 landing page — against the same hours one week earlier.
CVR was at week-over-week parity before the shutoff (+6%), after the shutoff (+4%), and this morning (roughly flat). Turning off the tests has not measurably moved sitewide conversion in either direction.
Rise-2 ran +22% above last week on its final day with tests on, and is tracking ~18% below parity this morning — on only 19 orders, which is noise territory. If the gap holds on clean data, the test may have been helping. Verdict: tomorrow.
The scary morning-of numbers ("CVR down 25%") were a measurement artifact, not a real drop. Google Analytics overcounts same-day sessions by ~35% until overnight processing. Details below.
The main paid-traffic landing page (~5,600 sessions/day). Yesterday was its strongest day vs. baseline in the window — with tests still running.
| Window | Sessions | Orders | CVR | Same hours last week | Δ |
|---|---|---|---|---|---|
| Wed Jun 11 full day — tests on until 7pm | 5,644 | 87 | 1.54% | 1.26% | +22% |
| Thu Jun 12, 12am–10am adjusted intraday | ~2,280 | 19 | ~0.83% | 1.01% | −18% |
The expectation was that removing test overhead would lift rise-2 conversion. The early data points the other way: the page outperformed its weekly baseline while a test was live, and is underperforming since. With only 19 orders this morning this is far from proven — but if it holds through today's full data, the killed rise-2 test was likely a winner and is worth restoring as the default experience.
Sitewide CVR has held week-over-week parity through the entire window — before the shutoff, after it, and this morning. Meta spend was flat (~$24–25k/day) all week, so media isn't confounding the comparison.
| Window | Sessions | Orders | CVR | Same hours last week | Δ |
|---|---|---|---|---|---|
| Wed Jun 11, 6am–7pm — tests ON | 18,375 | 340 | 1.85% | 1.74% | +6% |
| Wed Jun 11, 7pm–midnight — tests OFF | 5,003 | 74 | 1.48% | 1.41% | +4% |
| Thu Jun 12, 12am–10am adjusted intraday | ~9,790 | 157 | ~1.60% | 1.68% | −5% · noise |
The first pull of this data (6am Thursday) showed CVR running 20–25% below last week and a midnight explosion of "Unassigned" traffic. Both were artifacts of how Google Analytics processes data.
Same-day GA4 data is a rough draft. Google doesn't finish de-duplicating sessions or assigning traffic sources until an overnight batch job. Until then, session counts read ~35% too high (we verified directly: yesterday's 6am hour showed 2,117 sessions in the morning pull, and 1,571 after processing) and 35–40% of traffic shows as "Unassigned."
Inflated sessions mechanically deflate CVR. Same orders ÷ more phantom sessions = a conversion rate that looks 25–35% worse than reality. Comparing today's rough draft against last week's final data will always trigger a false alarm.
House rule going forward: intraday, trust order counts (those are accurate in real time). Judge CVR only on days that have finished processing. Orders this morning: 157 vs. 175 same hours last week — down 10%, within normal daily wobble.
Source: GA4 (property 310751693), sessions and transactions by hour, Pacific time. Jun 11 and baseline days (Jun 4–5) are fully processed; Jun 12 is intraday with sessions deflated by the measured 1.36× inflation factor. Intelligems' own analytics API was returning errors at time of writing, so GA4 is the source of record here. Meta spend cross-checked via Ads API: $24.5k (Jun 5) → $25.4k (Jun 11), flat; Jun 12 pacing normally.
Next step: re-run both cuts on fully processed Jun 12 data (Friday morning) for the final rise-2 verdict. If rise-2 is still below parity, pull the killed rise-2 experiment's results from Intelligems and consider shipping its variant as the default.