Independent product experiment · November 5, 2026
Turning a working intake flow into a measurable product decision
Raasta combines a human-reviewed quote workflow, two usable intake experiences and a local measurement loop. This case study explains the decision, the instrumented behavior and what the evidence can support.
Try Raasta ↗ · Inspect the source ↗ · Read the experiment plan ↗
The decision and my role
Should an intake form show every field at once or guide the participant through a short sequence? I treated cognitive load as an unvalidated usability risk and built a way to investigate it. I scoped the product question, designed the variants and event contract, implemented the workflow with AI assistance, and defined guardrails and next validation steps. No customer interviews or adoption gains are claimed.
Two experiences, the same underlying rules
The control displays the request, service, quantity and property-height fields together. Guided intake separates request and service selection, property details, and a review screen. Both use the same fictional price catalog and preserve the same human-review gate. This keeps the experiment focused on the intake experience.
The real GrowthBook JavaScript SDK assigns a stable anonymous alias to an arm using a pinned experiment key, equal weights and hash version 2. A missing identifier is not enrollment. The public demo lets a reviewer try either assigned experience.
What success means
The primary metric is the share of eligible exposed participants who reach a valid quote-review state. The funnel counts each participant once and ignores completion events before exposure. Reaching quote review is distinct from human approval, scheduling, revenue or a completed service job.
For planning only, a 40% baseline, a 15 percentage-point minimum detectable effect, two-sided 5% significance and 80% power imply about 173 participants per arm under a standard two-proportion approximation. A provisional target of 200 per arm adds planning headroom. These are assumptions to revisit after a pilot, not observed conversion results.
Analytics with a narrow data contract
Recording begins only when the operator starts a local test session. Events identify exposure, intake start, reaching review, abandonment, validation errors and human decisions. The allowlist includes an anonymous alias, variation, a synthetic-data marker and optional elapsed milliseconds. It excludes request text, names, contact details and quote contents.
Events remain in browser memory until exported or explicitly sent for local verification. The actual PostHog Node SDK sends them to a temporary loopback collector. That collector independently checks delivery and payload contents. This verifies SDK transport; it does not constitute a deployed PostHog analytics backend.
Guardrails before a winner
- Preserve identical quote logic and the human-review requirement.
- Inspect validation errors, handoffs and time to review alongside completion.
- Separate synthetic practice from consented usability-test data.
- Agree eligibility, exclusions and a stopping rule before collecting randomized traffic.
- Use initial usability sessions to find comprehension problems before attempting a powered experiment.
What has actually been checked
- Actual GrowthBook bucketing is repeatable and exercises both arms.
- Tests cover missing identity, duplicate completions and exposure ordering.
- The actual PostHog SDK delivered all 300 events from a 100-participant synthetic simulation to the local collector.
- Event validation strips unapproved properties.
- The existing LangGraph approval/resume and idempotence checks remain in place.
These are implementation results. The simulation applies different outcome rules by design. Its conversion rates are authored examples, not evidence that guided intake improves a real product. No paid API, real customer traffic or business outcome was used to establish these checks.
The next product decision
Run consented usability sessions, inspect where participants hesitate, and revise the flow and instrumentation. Once traffic and baseline estimates are adequate, evaluate the predefined completion metric, uncertainty and guardrails at the planned horizon. The current evidence supports proceeding to that validation step.
Sources and ownership
Original prototype, interface, fixtures, experiment framing and adapters by Ali Raza, with AI-assisted implementation. Upstream engines: LangGraph, GrowthBook, and PostHog SDK. Recorded local SDK delivery report. This is an independent public prototype, separate from any employer’s private systems.