A/B Testing Platform

Scope

This is a reference design for randomized product experiments, not a claim about one vendor's private implementation. It separates four things that are often incorrectly collapsed: deterministic assignment, actual treatment exposure, outcome attribution, and statistical inference. Correctness depends on a declared randomization unit, immutable experiment versions, replayable event data, and an analysis policy fixed before results are inspected.

Invariants

1. A unit is assigned from a canonical byte representation of namespace, experiment id, randomization-unit id, salt, and hash version. Every SDK uses golden cross-language test vectors. The hash need only be deterministic and well distributed for the chosen extraction; a cryptographic hash is not statistically disqualified merely because it is cryptographic.
2. Allocation is represented by versioned bucket intervals. A traffic ramp extends or deliberately remaps intervals under an explicit migration policy; it never silently changes the salt or unit.
3. Eligibility is not exposure. Exposure is logged at the point where the treatment is actually activated, with event id, event time, assignment/config version, unit, and variant.
4. Exposure and outcome streams are keyed compatibly by experiment and randomization unit, deduplicated by event id, and joined in event time. Watermarks, attribution windows, late-data handling, identity changes, bots, and missing outcomes are analysis inputs, not hidden cleanup.
5. Stream checkpoints alone do not make an end-to-end pipeline exactly once. The input must be replayable and every sink transactional or idempotent.
6. Results are immutable, versioned artifacts containing the cohort snapshot, metric definition, estimator, exclusions, code version, and trust checks.

[CONCEPT]stream-processing

Components

Node	Responsibility
Product App / Experiment SDK	Evaluate a signed config and activate a variant. High-risk permissions are still enforced server-side.
Signed Config CDN / Experiment API / Registry	Publish immutable allocations and a short-TTL emergency denylist; preserve configuration history.
Collector / Event Log	Validate events, attach ingestion metadata, retain a replayable ordered stream.
Event-Time Processor	Re-key, deduplicate, join within declared windows, and route invalid or late records explicitly.
Deduplicated Facts	Analysis-ready exposure and outcome facts; never the only copy of raw input.
Statistics Worker / Results Store	Reproducible estimators, sample-ratio checks, multiplicity policy, and audited results.
Guardrail Monitor	Applies a preregistered sequential rule with persistence/hysteresis; it does not repeatedly peek with an ordinary fixed-horizon p-value.

Scenario semantics

Deterministic assignment

The config is signed and cached. The SDK hashes a canonical tuple and maps it to a versioned bucket interval. A rollout operation must specify whether existing assignments are preserved. Assignment is stable only within the documented experiment version and identity policy; anonymous-to-authenticated identity transitions need an explicit rule.

Exposure and outcome

The exposure record means treatment activation, not a configuration lookup. Outcomes and exposures meet on the same experiment/unit key or on a durable assignment id. A first-qualifying-exposure policy avoids quietly changing cohorts when an SDK emits several exposures. Event-time intervals prevent an outcome from being joined merely because it arrived near an exposure in processing time.

Fixed-horizon calculation

The displayed numbers are an illustrative, internally coherent load and power example, not production telemetry.

For a two-sided test of proportions, a rough planning approximation is:

n per arm ≈ 2 × p × (1 − p) × (z(alpha/2) + z(power))² / delta².

With baseline p = 0.10, minimum detectable difference delta = 0.0015 (0.15 percentage points), alpha = 0.05, and 80% power, this gives about 628,000 units per arm before attrition or clustering adjustments.

The animation freezes 700,000 units per arm with control 10.00% and treatment 10.18%. The unpooled standard error is:

sqrt(0.1000 × 0.9000 / 700000 + 0.1018 × 0.8982 / 700000) = 0.000509,

or 0.0509 percentage points. The 0.18 percentage-point effect therefore has z ≈ 3.54, two-sided p ≈ 0.0004, and an ordinary fixed-snapshot 95% interval of approximately [0.080, 0.280] percentage points. This ordinary interval is not automatically valid after optional stopping. Continuous decisions need an always-valid or otherwise sequentially calibrated design.

CUPED-style variance reduction is dataset-dependent. In the idealized single-covariate case its variance factor is related to (1 − rho²); there is no universal 30–50% gain.

SRM and multiple comparisons

Sample-ratio mismatch compares observed assignment counts with the configured allocation. An exact 25/25/25/25 split has chi-square zero and p = 1, so the diagram instead uses an explicitly illustrative chi-square of 3.2 with 3 degrees of freedom (p about 0.36).

The multiple-testing family must be declared. Four arms create six all-pairs contrasts, but only three treatment-versus-control contrasts. Bonferroni/Holm control family-wise error; Benjamini–Hochberg targets false discovery rate. They are different guarantees, not interchangeable decorations.

Guardrail stop

A stop is triggered only by the preregistered safety rule and a sustained window, then audited. The control plane stops new assignment and publishes a signed denylist. Connected server SDKs or polling clients converge within their configured refresh behavior; an offline device cannot be promised a five-second push. Safety-critical access therefore remains server-authoritative.

Capacity model

No request volume is presented as an industry fact. For a deployment-specific worksheet:

• assignment reads per second = active clients × config refreshes per client per second, reduced by CDN hit rate;
• event ingress bytes per second = events per second × encoded event bytes × replication overhead;
• state retained by the join ≈ keyed events per second × attribution-window seconds × state bytes per event, after accounting for compaction and lateness;
• analysis scan cost = cohort rows × selected columns, or the cost of maintained sufficient statistics when their assumptions hold.

Measure skew by experiment and randomization unit. A single global launch can dominate a partition even when average throughput is comfortable.

Failure and privacy model

• A config-signature or schema failure serves control and reports telemetry; it does not invent a bucket.
• Collector retries use stable event ids. Poison records enter quarantine with a reason rather than blocking the partition forever.
• Checkpoint restore is tested together with replayable input and idempotent sinks.
• The raw archive has retention, access control, deletion, and regional policies. Hashing an account id is pseudonymization, not anonymization.
• Clock skew and offline uploads are carried as event metadata; analysis uses a documented timestamp policy.
• A statistics job never overwrites a published result. A new estimator or cohort creates a new result version.

What the diagram does not claim

It does not claim that a particular hash family is always unbiased, that ordinary p-values survive repeated peeking, that CUPED has a fixed improvement, or that a kill switch reaches disconnected clients instantly. It also does not turn an observational dashboard slice into a randomized conclusion.

References

• LaunchDarkly: Percentage rollouts — official example of deterministic context hashing and bucket allocations.
• Statsig: Assignment sources — required assignment-source fields and unit semantics.
• Statsig: Experiment filters — official guidance to log exposure at the actual exposure point when assignment logging is too early.
• Apache Flink: Fault tolerance — checkpoints, replayable sources, and end-to-end sink conditions.
• Apache Flink: Time and watermarks — event-time processing and late data.
• Microsoft Research: Diagnosing sample-ratio mismatch — SRM invalidates trust in an experiment.
• Microsoft Research: Deep dive into variance reduction — CUPED mechanisms and assumptions.
• Johari, Pekelis, Walsh: Always Valid Inference — primary paper on valid inference under continuous monitoring.

[CONCEPT]partitioning-strategies

Interactive scenarios

▶Walkthrough: deterministic-assignment (not found) ▶Walkthrough: exposure-and-outcome (not found) ▶Walkthrough: late-event-replay (not found) ▶Walkthrough: fixed-horizon-analysis (not found) ▶Walkthrough: srm-and-multiplicity (not found) ▶Walkthrough: guardrail-stop (not found)