Discovery Plan: Decision confidence before expansion

Purpose and status

Northstar is fictional. This is a future discovery plan, not a record of findings. It asks whether a product or analytics lead can use a compact, traceable view to choose an investigation without mistaking a rate for a conclusion.

Recruit five B2B SaaS product or analytics leads who regularly prepare metric reviews. Use fictional data only. Before the session, explain that participation is voluntary, the prototype is not a work-performance assessment, notes will be used only for the portfolio evaluation, and a participant can skip a task or stop without consequence. A participant may withdraw and request removal of their session data; record the withdrawal transparently without retaining or scoring removed data. Do not collect customer data, production metrics, or an opinion about Northstar as a real business.

Neutral comparison setup

Compare two presentation conditions with the same task prompts and reference access, using the two predefined fictional cases below:

A compact headline KPI view with rates and period labels, a separate local source-count/date reference and glossary, and a plain local text field. It has the same underlying information, but no inline source-count strip or scoped-note control.
The Metric Dashboard view, which exposes counts, definitions, cohort and observation dates, the unavailable state, and a scoped local note.

The comparison is not intended to establish that either design is superior. Counterbalance order: participants 1, 3, and 5 begin with the headline-only view; participants 2 and 4 begin with the traceable view. Use the other predefined fictional case in the second condition so a remembered answer does not substitute for interpretation; the cases are not assumed to have equal difficulty. Do not describe either condition as the preferred design before tasks begin.

Prepare the comparison before recruiting

The compact view is a study material to prepare, not a shipped dashboard mode. Use the committed fixture revision for both conditions; retain its revision and the two materials with the session notes. Give both conditions access to the same counts, definitions, cohort dates, observation dates, and fictional-data label. Otherwise the exercise would test withholding information instead of whether inline context is easier to use.

Prepare case A from 30 days / SMB (154/240 activation) and case B from 90 days / Enterprise (139/176 activation). These are different cases, not empirically matched difficulty levels. Assign cases and condition order before sessions: P1 A-compact/B-dashboard; P2 A-dashboard/B-compact; P3 B-compact/A-dashboard; P4 B-dashboard/A-compact; P5 A-compact/B-dashboard. Record this small, imperfect balance; do not claim a controlled performance effect from five sessions. Keep the exact same task wording and reference access in both conditions. Report errors by case and condition so case difficulty and order remain visible.

Pilot the script with an internal dry run to check that references are accessible and downloads work. An internal rehearsal is a software/materials check, not one of the five participant sessions. If materials change after a participant session, record the version and do not pool the two versions as identical conditions.

Consistent task script

For each condition, ask the participant to:

Explain one displayed rate and the population it represents.
Compare two segments and explain what the comparison supports.
Identify the current cohort and observation date relevant to the view.
Select what they would investigate next and state what additional evidence they would seek.
Prepare a scoped handoff. In the traceable condition, save an interpretation and export or copy the summary; in the headline-only condition, write the same handoff in a provided local text field.

Then show the Early access scenario in the traceable view and ask whether it represents 0% performance or unavailable observations. Keep facilitator prompts neutral: “What do you see?” and “What would you need before deciding?” Do not coach numerator, denominator, or desired wording.

What changes the direction

Use the measurement plan and validation protocol for denominators and withdrawal handling. Keep unsuccessful retained-consent attempts in the analysis. Reconsider the current explanation if participants miss the denominator, cohort lag, fictional-data disclosure, or unavailable state. If core comprehension is adequate but the scoped handoff remains difficult, test a lighter handoff before expanding notebook features. If the traceable view adds time without clearer explanations, reduce the density before adding metrics.