17 — A/B Testing and Experimentation

PRD Document · Savoy Signature Hotels — Multi-Site Headless Platform
Version: 1.0 · Date: 2026-03-04
Related docs: 16_Analytics_and_Tracking.md, 09_Cache_and_Performance.md

1. Purpose

This document outlines a conceptual approach for A/B testing and experimentation on the Savoy Signature platform. It serves as a reference for future evaluation.

[!CAUTION] This feature is experimental and is deferred to post-launch (Q4 2026). See 20_Roadmap_and_Phases.md for timeline. It should not be developed, estimated, or planned as part of the initial project scope. Implementation will only be considered post-launch, if and when the business case justifies it.

2. A/B Testing Strategy

2.1 Constraints

The headless architecture with aggressive Cloudflare edge caching creates a specific challenge for A/B testing: the same URL must serve different variants, but the edge cache serves a single version per URL.

2.2 Recommended Approach: Client-Side Experimentation

Given the Cloudflare Pro setup and the caching strategy, client-side experimentation is the most practical approach if experimentation is ever adopted.

Approach	Pros	Cons	Recommendation
Client-side (GTM/GA4)	No cache issues, no infra changes, fast to deploy	Possible flicker on variant swap	✅ First option (post-launch)
Edge-based (Cloudflare Workers)	No flicker, true server-side split	Requires Workers (separate billing), complex cache key management	⏳ Future evaluation
Server-side (Next.js middleware)	Full control, no flicker	Breaks edge cache (every request hits origin), kills TTFB	❌ Not recommended

3. Option A: Client-Side via GTM + GA4 (Recommended First Step)

3.1 How It Works

GTM assigns the user to a variant (A or B) using a random cookie or Math.random().
GTM injects the variant-specific changes (e.g., swapping a CTA label, changing a hero image URL) via DOM manipulation or CSS overrides.
GA4 receives the variant assignment as a custom dimension.
GA4 Explorations compare conversion rates between variant groups.

3.2 Anti-Flicker Snippet

To prevent the original content from flashing before the variant is applied:

<!-- Inline in <head>, before GTM -->
<style>.ab-hide { opacity: 0 !important; }</style>
<script>
  document.documentElement.classList.add('ab-hide');
  setTimeout(function() {
    document.documentElement.classList.remove('ab-hide');
  }, 500); // Fallback: show original after 500ms if GTM is slow
</script>

GTM removes the .ab-hide class after applying the variant, ensuring the correct version is displayed without flicker.

3.3 Experiment Scope

Client-side A/B testing is suitable for:

✅ Suitable	❌ Not Suitable
CTA button labels and colors	Entirely different page layouts
Hero image swaps	Server-rendered content changes
Copy changes (headings, descriptions)	URL-based routing experiments
Showing/hiding a module	Personalized content per user segment

4. Option B: Edge-Based via Cloudflare Workers (Advanced)

If experimentation needs grow beyond simple UI tweaks, a Cloudflare Workers-based solution can be evaluated.

4.1 Concept

The Worker inspects a cookie (ab_variant) and routes to variant-specific cache keys.
Each variant has its own cached version at the edge.
Trade-off: Doubles cache storage per experiment. Must be used sparingly.

[!NOTE] This approach is deferred to post-launch (Q4 2026). See 20_Roadmap_and_Phases.md for timeline. It will require a separate cost and feasibility analysis for Cloudflare Workers integration with the Pro tier.

5. Measurement and Reporting

5.1 GA4 Custom Dimensions

Dimension	Scope	Example Value
`experiment_id`	Session	`exp_hero_cta_2026Q2`
`experiment_variant`	Session	`A` or `B`

5.2 Minimum Sample Size

Before declaring a winner, experiments must reach statistical significance:

Minimum 1,000 sessions per variant (recommended baseline).
Minimum 2-week run time to account for day-of-week effects.
Use GA4’s built-in statistical analysis or external tools (e.g., Optimizely Stats Engine, AB Test Calculator).

6. Acceptance Criteria

GTM-based A/B test can be deployed without code changes (tag + trigger configuration only).
Anti-flicker snippet prevents visible content swap on variant assignment.
GA4 custom dimensions correctly segment traffic by experiment and variant.
Edge cache hit ratio remains > 95% during active experiments.
Experiments do not degrade Core Web Vitals (LCP, CLS, INP).
Test results are analyzed with statistical significance before any permanent change is applied.

Next document: 18_QA_Pipeline_and_Testing.md