Evaluation plan — proposed, not measured
Stage 1: five formative sessions
Recruit five consenting product operations or engineering-planning practitioners. Give every participant the same seeded Northstar conflict, then ask them to describe the constraint, make a draft change, compare it with the baseline, and decide whether to apply or undo it. Use four of five correct conflict explanations within 60 seconds as a diagnostic learning target only when five consented, started, non-withdrawn records are available; it is not a population estimate or a claim of research results. Record task time, help requested, incorrect assumptions, recovery behavior, confidence, and the participant's description of their current spreadsheet or meeting method.
The leading signals are recurring confusion about availability arithmetic, treating a draft as committed, or treating initiative risk as a delivery prediction. Iterate on the explanation before a later threshold study if any of those misunderstandings recur. Research has not occurred.
Stage 2: later ten-person task study
| Question | Canonical measure and denominator | Proposed threshold | Guardrail |
|---|---|---|---|
| Is the draft boundary understood? | Participants who edit, distinguish draft from baseline, and deliberately apply without help / all 10 consented, started, non-withdrawn task records | 9/10 | Assisted and abandoned attempts stay in the denominator and are reported separately. |
| Does the scenario expose unavailable assigned work? | Known zero-availability assignments surfaced / all known zero-availability assignments in the fixed scenario | 100% | This tests sample arithmetic, not productivity, burnout, or a delivery forecast. |
Use the same task order for the first half of participants and counterbalance whether the current spreadsheet or meeting method is discussed before or after the sample for the second half. Do not prime participants by calling the interface clearer, safer, or more accurate. The comparison is qualitative until a separately designed study defines a valid baseline.
Consent withdrawal is not an unsuccessful task. If a participant withdraws permission, delete their study material and report only the withdrawal count and study phase, without retaining their performance. Do not silently replace a withdrawn participant or remove an assisted or abandoned attempt. A 10-person threshold result requires 10 consented, started, non-withdrawn task records under the same protocol; otherwise treat the run as formative and repeat a complete cohort before making a threshold decision.
Automated checks test arithmetic and controls, not market demand. Automatic staffing recommendations remain rejected because they obscure uncertainty; real delivery estimates still require the appropriate planning and engineering discussion.