Learning & Expertise
In complex professional software, learning can continue for months or years. Usage data can identify changes in the methods people use and suggest where to investigate. It does not directly measure understanding, skill or the quality of their work.
For design approaches, see Designing for the Expertise Curve. To check whether a change represents learning, use realistic tasks, transfer and later sessions.
Learning Curve Mapping (Concept-Weighted)
Start with the concepts required for a representative task. Assigning simple weights can help a team discuss where the task introduces several unfamiliar ideas at once. The weights below are proposed design aids, not a validated difficulty scale.
| Weight | Working description | Examples to examine in context |
|---|---|---|
| 1 | A familiar, visible, single-purpose operation | Moving a shape or changing its colour |
| 2 | Understanding a relationship or mode | Masking, grouping or an animation behaviour |
| 3 | Understanding a system or abstraction | Constraints, expressions or procedural generation |
An experienced programmer may find an expression easier than a painter does. A mask can be simple or difficult depending on its relationships and feedback. Record those assumptions beside the weights.
Plot the cumulative demands of the task sequence to find places worth testing. This is a map of what the design asks people to understand, not an observed learning curve. Compare it with what participants can explain and accomplish independently. See Conceptual Burden.
Relate the Map to Usage
Track first use and subsequent use of relevant operations, with task and role where available. Repeated use may indicate familiarity, experimentation or continued difficulty. There is no established threshold at which three uses prove a concept has been learned.
A cluster of users who never reach a feature may suggest a discovery problem, a concept barrier or no need for it. Investigate the alternatives with people whose work would benefit from the feature.
The Power Law of Practice describes performance changes with practice under particular task conditions.1 It does not require a cumulative feature-use score to follow a particular shape, nor make deviations proof of a design fault.
Feature Discovery & Adoption
Helps detect: Visibility (features not being found), Conceptual Burden (features too complex to adopt). Both views below require the same foundational instrumentation — first-use tracking per feature per user — and together answer: which features are users reaching, and when?
Feature discovery and adoption are two complementary views of the same data. Adoption breadth tells you what proportion of users reach each feature. Discovery timeline tells you when in the user lifecycle they get there. The distinction suggests questions, rather than diagnoses. Early discovery with low continued use may reflect difficulty, limited usefulness or a specialised need. Late discovery may reflect poor visibility or a task that arises only later.
Adoption Breadth
What It Measures
The percentage of users who adopt each major feature, ordered from most to least adopted, visualised as a funnel.
Why It Matters
In complex software, unused features represent both wasted development effort and inaccessible value. The adoption funnel reveals the natural discovery order and identifies where significant drop-off occurs — features users could benefit from but aren't reaching.
How to Implement
Define feature tiers from basic to advanced, then track first-use per feature per user:
What to Watch For
Sharp drop-offs between adjacent features warrant investigation when the same users have reason and opportunity to use both. If 45% of users adopt animation but only 30% adopt effects, investigate:
-
Discovery problem? Users don't know effects exist → improve Visibility
-
Learning problem? Users can't figure out how effects work → reduce Conceptual Burden at that tier
-
Usefulness problem? Users don't need effects for their work → possibly expected, not a problem
The answer determines the appropriate change. Low adoption is not necessarily a problem: some features are specialised.
The overall funnel hides important patterns. A difference between new and established users may reflect discovery, changing tasks or role mix. Compare people with a demonstrated need for the feature before deciding that low adoption is a problem.
Discovery Timeline
What It Measures
How quickly users discover and first use features relative to their session count. For each feature: what's the median session number at which users first engage with it?
Why It Matters
Users should discover features in complex professional software progressively. Presenting every feature at once overwhelms users; keeping features hidden prevents discovery. Continued discovery can matter to users. Kujala and colleagues elicited retrospective accounts of changing experience from mobile-phone owners; these do not establish a required discovery trajectory for professional software.2
How to Implement
For each feature, record the session number at which each user first used it. Aggregate into a discovery timeline:
| Feature | Median Discovery Session | 25th Percentile | 75th Percentile |
|---|---|---|---|
| Basic Shapes | 1 | 1 | 1 |
| Text | 1 | 1 | 2 |
| Animation Behaviours | 4 | 2 | 12 |
| Effects | 8 | 3 | 25 |
| Render Queue | 12 | 5 | 40 |
| Scripting | 35 | 15 | 80+ |
What to Watch For
-
Very high median discovery session (e.g., 80+): The feature may be too hidden. The question is whether that's intentional (advanced features should take time to discover) or accidental (it's buried in a menu nobody opens).
-
Wide spread between 25th and 75th percentile: High variance means some users find it quickly while others take far longer. Check differences in roles, needs and opportunities before attributing this to chance.
-
Features never discovered by a segment that would benefit from them: A Visibility failure. Identify a task for which the feature would help, then check whether users can find and apply it.
Performance Dip Tracking
Helps detect: Rigidity (the cost of changing methods — deep dips mean transitions are punishing), Premature Commitment (high reversion rates mean users become locked into their initial, less efficient approach).
What It Measures
Temporary performance decreases when users move from a familiar, slower interaction method to a new, faster one, and whether they persist through the dip or revert.
Why It Matters
Scarr et al. (2011, Dips and Ceilings) demonstrated that users experience measurable performance dips when adopting more efficient methods — switching from menu navigation to keyboard shortcuts, for instance. A temporary loss of performance can discourage adoption. Its depth, duration and eventual benefit depend on the method, task and opportunity to practise.
This is the Paradox of the Active User in action: users prefer familiar but inefficient methods because the short-term cost of switching feels higher than the long-term benefit.
How to Implement
When a user first employs a new interaction method for an operation they previously performed differently:
- Record the transition event (e.g., "user performed [duplicate] via keyboard shortcut for the first time; previously used menu")
- Track their task completion time for that operation over subsequent sessions
- Compare with their average time using the prior method
Plot the result:
Time per operation
│
│ ╭─── Old method (stable but slow)
│ │
│ │ ╭── Dip (new method, initially slower)
│ │ ╱ ╲
│ │ ╱ ╲_____ New method (faster after ~5 sessions)
│ │ ╱
│──┼──────────────────────────
│ Sessions after transition
What to Watch For
-
Dip depth: How much slower is the user during the transition? Compare the cost with the benefit and the user's circumstances; 50% is not an established reversion threshold.
-
Dip duration: How many sessions until the new method becomes faster than the old? There is no universal five- or ten-session limit; measure comparable tasks and the practice actually available.
-
Reversion rate: What percentage of users who try the new method revert? High reversion rates suggest the temporary loss of performance is too costly or the long-term benefit is not sufficiently clear.
-
Design interventions: Consider "you're getting faster" feedback, progress indicators, or side-by-side timing comparisons to help users persist through the dip.
Skill Progression Curves
Can suggest questions about Conceptual Burden and Consistency. A flat score or a wide percentile band can also reflect task mix, prior experience or the scoring system.
What It Measures
Aggregate sophistication score trajectories across your user base, plotted by "week of life" (week 1 = the user's first seven days, week 2 = days 8-14, etc.), showing percentile bands (P25/P50/P75) to reveal both the typical progression and its variance.
Why It Matters
Individual usage scores summarise recorded behaviour. Their trajectories show changes in that behaviour; call them skill progression only if you have established a relationship with independent task performance.
Practice can improve performance, but a feature-use score is not a direct measure of that improvement.1 A stable score can accompany faster, better work within a narrow repertoire. A falling score can accompany a simpler project.
How to Implement
For all users with sufficient tenure (4+ weeks of activity), calculate their sophistication score at each week of life. Aggregate across users to produce percentile bands:
| Week of Life | P25 Score | Median Score | P75 Score | Users in Cohort |
|---|---|---|---|---|
| Week 1 | 3 | 6 | 12 | 450 |
| Week 2 | 5 | 9 | 16 | 380 |
| Week 4 | 8 | 14 | 22 | 290 |
| Week 8 | 10 | 18 | 28 | 180 |
| Week 12 | 12 | 21 | 32 | 120 |
Plot as a band chart with the P25-P75 range shaded and the median as a solid line. The numbers and twelve-week window are illustrative. Report eligible users and changing cohort composition: later percentiles may describe the people who remained, rather than improvement by the original cohort.
What to Watch For
-
Flat P25 line: The bottom quartile has a stable recorded score. Check whether comparable task performance is also stable before concluding that users are not developing capability — the Paradox of the Active User in aggregate form. Investigate what features they're missing.
-
Wide P25-P75 bands: High variance may reflect different tasks, prior experience or opportunities to practise. Some users progress rapidly while others stall. Check whether the split correlates with specific feature discovery or workflow patterns.
-
Convergence at the top: If P50 and P75 converge, there may be a ceiling effect — either in your scoring system or in the software's actual capability depth. Check whether power users are hitting the limits of what the tool can do.
-
Regression at specific weeks: A dip in median score at a particular week may correlate with a version change that disrupted established workflows.
Plateau and Regression Detection
Helps detect: Rigidity (plateaued users may find the cost of learning new approaches too high), Conceptual Burden (ceiling effects at specific sophistication bands), Consistency (version-induced regressions indicate a design change broke established patterns).
What It Measures
Users whose recorded usage score is stable or falling. These are candidates for investigation. A score plateau is not necessarily a skill plateau, and a score decline is not necessarily lost capability.
Why It Matters
A stable score may help locate users worth talking to about the Paradox of the Active User. Confirm that their present method is limiting relevant work before recommending a different one.
Regressions provide an even clearer basis for action. If a user's sophistication score drops after a version update, the update may have changed or broken a workflow they relied on. They may, for instance, stop using a feature they previously used regularly. Version-induced regressions are a serious quality signal that won't appear in crash reports or error logs.
How to Implement
The periods and point thresholds below are examples to calibrate, not validated diagnostic cut-offs.
Stable-score detection:
-
Compare a user's sophistication score across their last 3-4 measurement periods
-
Flag users whose score hasn't changed by more than ±2 points across all periods as candidates for investigation
-
Segment candidates by score band, task and opportunity to use relevant features
Regression detection:
-
Compare a user's current-period sophistication score to their historical peak
-
Flag users whose current score is 3+ points below their peak as candidates for a regression check
-
Cross-reference regression timing with version changes — did the drop coincide with a software update?
| Segment | Plateaued Users | % of Segment | Avg Weeks Since Growth |
|---|---|---|---|
| Score 0–9 | 45 | 62% | 8.2 |
| Score 10–24 | 38 | 41% | 6.1 |
| Score 25–39 | 12 | 18% | 4.5 |
What to Watch For
-
Many stable scores in one band: check what these users are trying to do. A common unmet need may expose a barrier; a common narrow role may explain the stability.
-
Regression clusters after a software update: Multiple users regressing simultaneously, correlated with a version change, may indicate a disrupted workflow; check instrumentation changes and task mix too. Investigate which features stopped being used.
-
Advanced users regressing: Power users who regress may indicate performance problems at scale (their projects outgrew the tool), competitive loss (they found a better tool for their advanced needs), or workflow disruption from an update.
-
New users with stable scores: investigate independent task success and unmet needs. Some may have reached everything they need; others may be unable to proceed.