Learning & Expertise
For complex professional software, the learning curve is the product experience — not for days, but for months or years. These analytics track whether users are actually progressing, where they get stuck, and what predicts successful expertise development. They're not onboarding diagnostics; they're ongoing health indicators.
For what to actually build to close the gaps these metrics reveal — training-wheels interfaces, rehearsal paths, protected friction — see Designing for the Expertise Curve.
Three metrics in this section measure sophistication progression at different scales — use them together for a complete picture:
- Learning Curve Mapping — what users need to learn (concept-weighted curriculum and where cliffs exist)
- Skill Progression Curves — how fast users learn it (aggregate trajectories with percentile bands)
- Plateau and Regression Detection — where individual users stop learning (individual-level diagnostics)
Learning Curve Mapping (Concept-Weighted)
Directly measures: Conceptual Burden (the conceptual gradient from live data). Learning cliffs in the observed curve pinpoint exactly where concept weight spikes — your highest-priority redesign targets for reducing conceptual burden.
What It Measures
Your software's actual learning curve, observed through telemetry, by mapping concept usage against workflow complexity. This extends the Conceptual Gradient framework from Usability Harmonics into measurable, ongoing analytics.
Why It Matters
The Conceptual Burden section of Usability Harmonics describes how to evaluate a learning curve by assigning weights to concepts and plotting them against workflow difficulty. This metric turns that theoretical evaluation into a live measurement — you can observe which concepts users have learned, how quickly they learn them, and exactly where the curve steepens into a cliff.
How to Implement
Step 1: Weight your concepts (from your Conceptual Design)
| Weight | Criteria | Examples |
|---|---|---|
| 1 (Simple) | Single-purpose, intuitive, visible | Basic shapes, colour picker, move tool |
| 2 (Moderate) | Requires understanding relationships or modes | Animation behaviours, masking, grouping |
| 3 (Complex) | Requires understanding systems, composition, or abstraction | Scripting, constraint systems, procedural generation |
Step 2: Define workflow difficulty levels
Map your software's typical workflows to difficulty tiers based on cumulative concept weight:
| Level | Concept Weight Sum | Example Workflow |
|---|---|---|
| Basic | 1-5 | Create a simple shape, change its colour |
| Intermediate | 6-15 | Animate a shape with behaviours, add effects |
| Advanced | 16-30 | Build a complex composition with constraints and expressions |
| Expert | 31+ | Scripted procedural animation with custom rendering |
Step 3: Track concept vocabulary per user
For each user, maintain a running set of concepts they've engaged with. Their "concept vocabulary" is the sum of weights for all concepts they've used meaningfully (3+ uses is a reasonable threshold — once could be accidental).
Step 4: Plot the Conceptual Gradient from real data
The gap between "Ideal" and "Actual" reveals learning cliffs — places where concept weight jumps sharply between adjacent difficulty levels.
What to Watch For
-
User clustering at a concept weight threshold: If many users stall at a cumulative weight of ~15, there's a cliff between Intermediate and Advanced. The specific concepts at that threshold are your redesign targets.
-
Power Law shape: Newell & Rosenbloom's Power Law of Practice (1981)1 predicts learning follows a power curve — rapid early gains, slower later ones. Your observed curve should roughly match this shape. If it doesn't, the conceptual design has unnecessary complexity spikes.
-
Time-to-level by cohort: Track how long it takes different cohorts to reach each difficulty level. If later cohorts reach Intermediate faster than earlier ones, your onboarding improvements are working.
Feature Discovery & Adoption
Helps detect: Visibility (features not being found), Conceptual Burden (features too complex to adopt). Both views below require the same foundational instrumentation — first-use tracking per feature per user — and together answer: which features are users reaching, and when?
Feature discovery and adoption are two complementary views of the same data. Adoption breadth tells you what proportion of users reach each feature. Discovery timeline tells you when in the user lifecycle they get there. The distinction matters: a feature with low adoption but early discovery has a learning or usefulness problem; a feature with late discovery but high eventual adoption has a visibility problem that users overcome with time.
Adoption Breadth
What It Measures
The percentage of users who adopt each major feature, ordered from most to least adopted, visualised as a funnel.
Why It Matters
In complex software, unused features represent both wasted development effort and inaccessible value. The adoption funnel reveals the natural discovery order and identifies where significant drop-off occurs — features users could benefit from but aren't reaching.
How to Implement
Define feature tiers from basic to advanced, then track first-use per feature per user:
What to Watch For
Sharp drop-offs between adjacent features indicate discovery or accessibility problems. If 45% of users adopt animation but only 30% adopt effects, investigate:
-
Discovery problem? Users don't know effects exist → improve Visibility
-
Learning problem? Users can't figure out how effects work → reduce Conceptual Burden at that tier
-
Usefulness problem? Users don't need effects for their work → possibly expected, not a problem
The answer determines the fix. Don't assume all low adoption is a problem — some features are genuinely specialised.
The overall funnel hides important patterns. A feature adopted by 30% of all users but 85% of Veterans is healthy — it's an advanced capability being discovered by those who need it. A feature adopted by 30% of all users including Veterans is probably too hidden or too hard to use.
Discovery Timeline
What It Measures
How quickly users discover and first-use features relative to their session count. For each feature: what's the median session number at which users first engage with it?
Why It Matters
Features in complex professional software should be discovered progressively — not all at once (overwhelming) and not never (hidden). A healthy product shows steady feature discovery across the user's lifetime, creating what Kujala et al. (2011, UX Curve) identified as the "continuous discovery of deeper capabilities" that drives long-term professional tool satisfaction.2
How to Implement
For each feature, record the session number at which each user first used it. Aggregate into a discovery timeline:
| Feature | Median Discovery Session | 25th Percentile | 75th Percentile |
|---|---|---|---|
| Basic Shapes | 1 | 1 | 1 |
| Text | 1 | 1 | 2 |
| Animation Behaviours | 4 | 2 | 12 |
| Effects | 8 | 3 | 25 |
| Render Queue | 12 | 5 | 40 |
| Scripting | 35 | 15 | 80+ |
What to Watch For
-
Very high median discovery session (e.g., 80+): The feature may be too hidden. The question is whether that's intentional (advanced features should take time to discover) or accidental (it's buried in a menu nobody opens).
-
Wide spread between 25th and 75th percentile: High variance means some users find it quickly while others take far longer — discovery likely depends on stumbling upon it rather than a clear path.
-
Features never discovered by a segment that would benefit from them: A Visibility failure. Cross-reference with sophistication scores to identify users who should be using a feature based on their overall capability but aren't.
Performance Dip Tracking
Helps detect: Rigidity (the cost of changing methods — deep dips mean transitions are punishing), Premature Commitment (high reversion rates mean users become locked into their initial, less efficient approach).
What It Measures
Temporary performance decreases when users transition from a familiar (but slower) interaction method to a new (faster) one — and whether they persist through the dip or revert.
Why It Matters
Scarr et al. (2011, Dips and Ceilings) demonstrated that users experience measurable performance dips when adopting more efficient methods — switching from menu navigation to keyboard shortcuts, for instance. The dip feels like failure, and most users revert to the slower method, never realising the faster path would have paid off within a few sessions.
This is the Paradox of the Active User in action: users prefer familiar but inefficient methods because the short-term cost of switching feels higher than the long-term benefit.
How to Implement
When a user first employs a new interaction method for an operation they previously performed differently:
- Record the transition event (e.g., "user performed [duplicate] via keyboard shortcut for the first time; previously used menu")
- Track their task completion time for that operation over subsequent sessions
- Compare with their average time using the prior method
Plot the result:
Time per operation
│
│ ╭─── Old method (stable but slow)
│ │
│ │ ╭── Dip (new method, initially slower)
│ │ ╱ ╲
│ │ ╱ ╲_____ New method (faster after ~5 sessions)
│ │ ╱
│──┼──────────────────────────
│ Sessions after transition
What to Watch For
-
Dip depth: How much slower is the user during the transition? Dips exceeding 50% performance loss are likely to cause reversion.
-
Dip duration: How many sessions until the new method becomes faster than the old? More than 5-10 sessions is a long time to endure feeling slower.
-
Reversion rate: What percentage of users who try the new method revert? High reversion rates suggest the dip is too punishing or the long-term benefit isn't visible enough.
-
Design interventions: Consider "you're getting faster" feedback, progress indicators, or side-by-side timing comparisons to help users persist through the dip.
Skill Progression Curves
Helps detect: Conceptual Burden (slow progression indicates high burden), Consistency (wide P25-P75 bands suggest the learning experience is inconsistent across users). Flat median curves at specific weeks reveal where the conceptual design creates ceilings.
What It Measures
Aggregate sophistication score trajectories across your user base, plotted by "week of life" (week 1 = the user's first seven days, week 2 = days 8-14, etc.), showing percentile bands (P25/P50/P75) to reveal both the typical progression and its variance.
Why It Matters
Individual sophistication scores tell you where a user is now. Skill progression curves tell you whether your user base is learning at a healthy rate overall, and how much variance exists. This is the difference between a snapshot and a trajectory — and trajectories reveal structural problems that snapshots miss.
Newell & Rosenbloom's Power Law of Practice (1981)1 predicts that learning follows a power curve — rapid early gains, slower later ones. Your observed skill progression should roughly match this shape. Deviations reveal design problems: a flat curve means users aren't learning; an early plateau means there's a ceiling; a sudden drop means something disrupted established workflows.
How to Implement
For all users with sufficient tenure (4+ weeks of activity), calculate their sophistication score at each week of life. Aggregate across users to produce percentile bands:
| Week of Life | P25 Score | Median Score | P75 Score | Users in Cohort |
|---|---|---|---|---|
| Week 1 | 3 | 6 | 12 | 450 |
| Week 2 | 5 | 9 | 16 | 380 |
| Week 4 | 8 | 14 | 22 | 290 |
| Week 8 | 10 | 18 | 28 | 180 |
| Week 12 | 12 | 21 | 32 | 120 |
Plot as a band chart with the P25-P75 range shaded and the median as a solid line. Tracking weeks 1 through 12 provides sufficient resolution to identify progression patterns; the declining user count at later weeks naturally shows attrition alongside skill development.
What to Watch For
-
Flat P25 line: The bottom quartile never progresses. These users are active but not developing capability — the Paradox of the Active User in aggregate form. Investigate what features they're missing.
-
Wide P25-P75 bands: High variance means the learning experience is inconsistent. Some users progress rapidly while others stall. Look for whether the split correlates with specific feature discovery or workflow patterns.
-
Convergence at the top: If P50 and P75 converge, there may be a ceiling effect — either in your scoring system or in the software's actual capability depth. Check whether power users are hitting the limits of what the tool can do.
-
Regression at specific weeks: A dip in median score at a particular week may correlate with a version change that disrupted established workflows.
Plateau and Regression Detection
Helps detect: Rigidity (plateaued users may find the cost of learning new approaches too high), Conceptual Burden (ceiling effects at specific sophistication bands), Consistency (version-induced regressions indicate a design change broke established patterns).
What It Measures
Users whose sophistication score has stopped increasing for an extended period (plateau) or has decreased after a software update (regression). This is distinct from engagement decline — a user may be active daily but stuck at the same capability level, or may have lost capability they previously demonstrated.
Why It Matters
Plateaus represent the Paradox of the Active User made visible in data — users stuck at a capability level despite continued use. Detecting them tells you where in the sophistication spectrum users get stuck, which maps directly to concept barriers in your design.
Regressions are even more actionable. If a user's sophistication score drops after a version update — they stop using a feature they previously used regularly, say — the update may have changed or broken a workflow they relied on. Version-induced regressions are a serious quality signal that won't appear in crash reports or error logs.
How to Implement
Plateau detection:
-
Compare a user's sophistication score across their last 3-4 measurement periods
-
Flag users whose score hasn't changed by more than ±2 points across all periods as "plateaued"
-
Segment plateaued users by their plateau level (Beginner, Intermediate, Advanced)
Regression detection:
-
Compare a user's current-period sophistication score to their historical peak
-
Flag users whose current score is 3+ points below their peak as "regressed"
-
Cross-reference regression timing with version changes — did the drop coincide with a software update?
| Segment | Plateaued Users | % of Segment | Avg Weeks Since Growth |
|---|---|---|---|
| Beginner (0-9) | 45 | 62% | 8.2 |
| Intermediate (10-24) | 38 | 41% | 6.1 |
| Advanced (25-39) | 12 | 18% | 4.5 |
What to Watch For
-
High plateau rate at Intermediate level: The most common pattern — users learn the basics but hit a ceiling before reaching advanced capability. This is your highest-leverage design improvement target. Cross-reference with Feature Adoption to identify the specific features at the barrier.
-
Regression clusters after a software update: Multiple users regressing simultaneously, correlated with a version change, indicates the update disrupted established workflows. Investigate which features stopped being used.
-
Advanced users regressing: Power users who regress may indicate performance problems at scale (their projects outgrew the tool), competitive loss (they found a better tool for their advanced needs), or workflow disruption from an update.
-
Beginners plateauing immediately: Users who plateau in their first few weeks and never progress likely hit the concept barrier too early. Cross-reference with First Session Quality and Churner Segmentation.