Measurement plan — proposed, not measured
Stage 1: five formative sessions
Recruit five consenting product operations or release leads. Use the same fictional Analytics 2.4 scenario for each participant: identify the initial blockers, prepare the missing sample information, record a decision, simulate the launch, respond to the sample issue, and explain the timeline. Do not turn this diagnostic sample into a population claim. Record time to identify blockers, help requested, incorrect launch attempts, misconceptions about evidence or simulation, abandonment, response choice, and five-point confidence.
Leading indicators are repeated confusion that a readiness percentage grants permission, that nonblank evidence is verified proof, or that the sample deploys software. Revise the blocker explanation or decision language before a threshold study if any misconception recurs. Research has not occurred.
Stage 2: later 20-person task study
| Question | Canonical measure and denominator | Proposed threshold | Guardrail |
|---|---|---|---|
| Can a reviewer explain why launch is blocked? | Correct blocker explanations / all 20 consented, started, non-withdrawn task records | 18/20 | The explanation names a missing prerequisite or required evidence and connects it to disabled launch; repeating a percentage alone does not pass. |
| Can a reviewer complete the gated flow? | Unassisted completion from initial blockers through simulated launch / all 20 consented, started, non-withdrawn task records | 17/20 | Assisted and abandoned attempts remain in the denominator and are reported separately. |
| Does every started sample issue end with a response? | Recorded Pause rollout or Roll back release choices / all participants who start the sample-incident task | 100% | Report issue-task starts separately from gated-flow completion; the two response paths model only a fictional scenario. |
Use the same product tasks in the same order for every participant. Randomly assign 10 participants to describe their current method before the sample and 10 afterward; record the assignment but treat the comparison as qualitative until a study defines a valid baseline. Do not prime participants by calling the workspace safer, more accountable, or more reliable.
Consent withdrawal is not an unsuccessful task. If a participant withdraws permission, delete their study material and report only the withdrawal count and study phase. Do not silently replace a withdrawn participant or exclude assisted or abandoned attempts. A 20-person threshold result requires 20 consented, started, non-withdrawn records under the same protocol; otherwise treat the run as formative and repeat a complete cohort before making a threshold decision.
Stop expansion and return to the readiness explanation if participants treat text presence as truth, mistake simulated launch for a deployment, or cannot explain why a Go is withdrawn after a required regression. Do not propose templates, integrations, or real-data intake until those boundaries are understood.