Analytics for Usability Evaluation
Usability Harmonics gives you a structured way to evaluate complex software — but evaluations are periodic and manual. Between them, how do you know whether problems are emerging, interventions are working, or a new release has quietly broken something? Analytics provide continuous, automated signal between manual evaluations.
Analytics can't replace manual evaluation. They measure behaviour, not cognition — a high undo rate tells you users are struggling, not why. But they can:
- Prioritise which dimensions need attention before you run a full evaluation
- Validate whether design interventions actually moved the needle
- Detect regressions between evaluations, especially after releases
What follows describes how to use the analytics from the preceding sections as a companion to Usability Harmonics testing.
Pre-Evaluation Triage
Before running a full Usability Harmonics evaluation, review your analytics dashboard to identify which dimensions are most likely to have problems. This focuses evaluator attention where it matters — important because evaluator expertise matters far more than evaluation method choice (Hvannberg et al., 2007 found only 21% overall effectiveness in heuristic evaluation, with inter-evaluator reliability at a dismal 0.046).1
| If analytics show... | Investigate this dimension | Using this analytics type |
|---|---|---|
| Users plateau at intermediate sophistication levels | Conceptual Burden (concept cliff) | Sophistication Scoring, Skill Progression Curves |
| Low adoption for features at a specific tier | Visibility (features not found) | Feature Adoption Funnel, Feature Discovery Rate |
| High undo/error rates that persist across maturity levels | Error-Proneness (design failure, not training gap) | Friction Signals |
| Veterans use the same creation methods as novices | Visibility, Premature Commitment | Creation Method Distribution, Performance Dip Tracking |
| Novice and expert workflow patterns differ radically | Diffuseness, Rigidity | Workflow Pattern Analysis |
| Users produce complex work but churn anyway | Hard Mental Operations, Rigidity | Project Complexity + Churner Segmentation |
| "Struggled" churner segment dominates | Conceptual Burden (concept barrier) | Churner Segmentation, First Session Quality |
| Regression clusters after software updates | Consistency | Plateau and Regression Detection |
| Slow Starters never accelerate | Conceptual Burden (intermediate cliff) | New User Onboarding Journeys |
| Wide P25-P75 bands in skill progression | Consistency (inconsistent learning experience) | Skill Progression Curves |
If your analytics point strongly to two or three dimensions, brief your evaluators to focus there first. Blackwell & Green's Cognitive Dimensions research shows that activity profiles determine which dimensions matter most — analytics serve as a proxy for identifying the dominant activity profiles before evaluation begins.
Post-Evaluation Validation
After making design changes based on a Usability Harmonics evaluation, specific analytics can validate impact. This closes the loop between qualitative evaluation and quantitative measurement.
Conceptual Burden Intervention
You've redesigned a feature to reduce conceptual weight — perhaps simplifying a multi-concept workflow into fewer concepts, or improving the conceptual gradient by reordering the learning path.
Track:
- Sophistication score distribution: Does it shift upward?
- Plateau band: Does the Intermediate plateau thin?
- Skill progression curves: Does the median curve steepen at the intervention point?
- Session milestone funnel: Does the drop-off at the targeted milestone decrease?
Timeline: Allow 4–8 weeks for the effect to appear in cohort data. Immediate changes in raw funnel numbers may reflect seasonal variation rather than your intervention.
Visibility Intervention
You've made a hidden feature more discoverable — surfaced it in context, added progressive disclosure cues, or restructured the interface hierarchy.
Track:
- Feature adoption rate: Does it increase for the target feature?
- Feature discovery session: Does the median session number decrease? (Users find it sooner.)
- Feature adoption funnel: Does the drop-off at the intervention point decrease?
Timeline: 2–4 weeks. Visibility changes show up quickly in discovery metrics.
Error-Proneness Intervention
You've redesigned a workflow to reduce errors — added constraints, improved feedback, or restructured the interaction sequence.
Track:
- Undo rate for the specific workflow: Does it decrease?
- Error encounter rate for the target error type: Does it decrease?
- Undo rate segmented by maturity: Does the error rate decrease for both new and veteran users? If it only drops for new users, the fix may not address the veteran workflow.
Timeline: 2–4 weeks. Error reduction should appear quickly if the fix is effective.
Rigidity Intervention
You've reduced the cost of changing approach — better undo/redo support, non-destructive editing, or easier method switching.
Track:
- Creation method distribution: Does it shift? (More users adopting the more flexible approach.)
- Performance dip reversion rate: Does it decrease? (Users persist through the transition more often.)
- Workflow pattern analysis: Do expert and novice patterns converge? (The efficient path becomes accessible earlier.)
Timeline: 4–8 weeks. Rigidity changes affect workflow habits, which evolve slowly.
Continuous Monitoring
Rather than reviewing every metric before each evaluation, maintain a small set of sentinel metrics on a weekly cadence. These provide early warning of dimension regressions without requiring a full dashboard review.
Sentinel Metrics
| Sentinel | What to Watch | Dimension | Alert Threshold |
|---|---|---|---|
| Sophistication shift | Median sophistication score across active users | Conceptual Burden | Drops 2+ points over 4 weeks |
| Discovery rate | Median discovery session for recently launched features | Visibility | Above 20 sessions for a feature intended for intermediate users |
| Veteran friction | Undo/error rate for users with 50+ sessions | Error-Proneness | Increases 15%+ over 4 weeks |
| Method stagnation | % of Regular/Veteran users still using novice methods | Premature Commitment | Above 60% for methods where an efficient alternative exists |
| Regression count | Users whose sophistication dropped 3+ points this period | Consistency | More than 5% of active users, especially after a release |
Reading the Sentinels
Sentinels are designed to be glanceable — you should be able to assess product health in under a minute. The key patterns:
- All sentinels stable: No immediate concerns. Continue with your planned evaluation cadence.
- One sentinel triggered: Investigate that specific dimension before the next evaluation. Likely a localised issue.
- Multiple sentinels triggered simultaneously: Something systemic has changed — a major release, a competitor shift, a platform change. Consider an ad-hoc evaluation focused on the affected dimensions.
- Sentinel triggered immediately after a release: High priority. The release may have introduced a regression. Cross-reference with Plateau and Regression Detection to identify affected users and features.
Aspirational: Activity Profile Detection
This analytics approach is more experimental than the proven patterns above. It requires more instrumentation effort and draws from the theoretical framework rather than production-validated implementations.
The Cognitive Dimensions framework (Blackwell et al., 2001) emphasises that the same software serves different activity types, and that different Usability Harmonics dimensions matter for each. Blackwell & Green found that programmers spent 22.3% of their time restructuring and 18% exploring, while musicians using the same notation software spent 51.5% transcribing and only 2.4% exploring.2 Same tool, radically different dimensional priorities.
This means aggregate analytics hide activity-specific problems. A high undo rate during exploratory design is healthy — the user is experimenting. The same undo rate during transcription is a problem — the user is making errors. If you can detect which activity a user is engaged in, you can segment every other metric by activity type, dramatically improving diagnostic value.
The Five Activity Types
The Cognitive Dimensions framework defines five core activities when working with notational systems:
| Activity | Behavioural Signals | Key UH Dimensions |
|---|---|---|
| Search | High navigation-to-action ratio, frequent view switches, minimal creation events | Visibility, Hard Mental Operations |
| Transcription | Steady creation rate, low undo, low exploration, linear progression | Consistency, Diffuseness |
| Incrementation | Small modifications to existing work, low creation rate, frequent property changes | Rigidity (viscosity), Error-Proneness |
| Modification | High selection-change rate, property edits on existing elements, moderate undo | Rigidity, Invisible Links |
| Exploratory Design | High creation-and-deletion rate, diverse feature usage, non-linear workflow, frequent undo | Premature Commitment, Provisionality |
How This Could Be Implemented
Classify session segments (say, five-minute windows) by the dominant activity pattern based on action-type ratios. The classification is inherently approximate, but even rough classification improves metric interpretation significantly.
With activity detection you could produce:
-
Friction Signals segmented by activity: "Undo rate during exploration is 12/hour (healthy). Undo rate during transcription is 8/hour (problematic — users are making errors during routine work)."
-
Feature discovery by activity: "Users in exploratory mode discover 3x more features per session than users in transcription mode — but only 15% of sessions are exploratory."
-
Time allocation across activities: "Expert users spend 40% of time in modification, 25% in exploration. Novice users spend 60% in search, 20% in transcription." This directly reveals that novices spend most of their time navigating rather than working.
What This Enables
Activity-aware analytics transform the triage table from Pre-Evaluation Triage. Instead of "high undo rate → investigate Error-Proneness", you get "high undo rate during transcription → investigate Error-Proneness for transcription workflows" — a much more targeted investigation.
Aspirational: Wayfinding Analytics
This analytics approach addresses a gap identified in Mirel's research on complex problem solving. It requires navigation-specific instrumentation that most applications don't yet provide.
Barbara Mirel's (2004) research on complex problem solving documented that wayfinding — navigating information landscapes, re-finding previous states, maintaining orientation — is the least designed-for activity in professional software, yet it consumes enormous time and cognitive effort. Users spend substantial portions of their sessions on what Mirel calls "enabling tasks" — work that supports the work but doesn't directly produce output. Wayfinding is one category of enabling task; the full scope is covered in the Enabling Task Analytics section below.
The arithmetic is compelling: if users spend 40% of their time navigating and 60% working, a 10% improvement in navigation efficiency frees more productive time than a 10% improvement in the working tools themselves.
What to Instrument
| Signal | What It Reveals | UH Dimension |
|---|---|---|
| View/panel switches per minute | Navigation intensity — high rates suggest users can't find what they need | Visibility |
| Search-to-find ratio | How many searches it takes to find a specific item | Visibility, Hard Mental Operations |
| Repeated searches for the same item | Persistent findability problems | Visibility |
| Bookmark creation and return rates | Whether users build and use their own navigation shortcuts | Secondary Notation |
| Undo-as-navigation | Using undo not to fix errors but to return to a previous state | Provisionality, Premature Commitment |
| Navigation ratio | Time spent navigating (view switches, searching, scrolling without acting) vs. time spent acting (creating, modifying, configuring) | Overall wayfinding health |
What Healthy Patterns Look Like
-
Navigation ratio decreases with maturity: New users spend more time navigating (learning the space); veterans spend more time working (they know where things are). If the ratio doesn't decrease, the information architecture has structural findability problems.
-
Low repeated-search rate: Users find things on the first or second try. High repeat rates suggest search doesn't match user mental models.
-
Bookmark usage correlates with productivity: Users who create navigation shortcuts produce more complex work. If bookmark usage is very low, the bookmarking system may be too hidden or too limited.
-
Undo-as-navigation is rare: Users have proper navigation tools — history, state management, bookmarks — and don't need to abuse undo to revisit previous states. High undo-as-navigation rates indicate the software lacks adequate state management.
Connection to the Four Micro Contextual Questions
Wayfinding analytics directly measure how well your software answers the four micro contextual questions:
- "What have I done up to this point?" — High undo-as-navigation suggests users can't review their history effectively.
- "What did I just do?" — Repeated identical actions suggest the software isn't providing clear feedback on what just happened.
- "What's the current state?" — High view-switching rates suggest users can't assess current state from their current view.
- "What can be done next?" — High search rates for operations suggest users can't see what actions are available in context.
Aspirational: Enabling Task Analytics
This analytics framework addresses a fundamental insight from Mirel's research that current analytics approaches largely ignore. It requires a conceptual shift in how events are classified, not just new instrumentation.
Barbara Mirel's (2004) research draws a crucial distinction between two types of work in complex software:
- Mainline tasks: Core operations that directly produce output — creating elements, editing properties, composing, animating, rendering, exporting. This is what users think of as "the work."
- Enabling tasks: Work that supports mainline tasks but doesn't directly modify the creative output — annotating, comparing versions, bookmarking positions, capturing rationale, organising workspaces, searching for assets, configuring settings, preparing data. Mirel describes these as tasks that "do not change the content and structure of the data for analytical purposes but are nonetheless essential to success."
Why Current Analytics Miss This
Standard feature-usage metrics count all actions equally. A session with 80% enabling tasks and 20% mainline tasks looks identical to a session with 20% enabling and 80% mainline — same total action count, same session duration, same "engagement." But these are completely different user experiences. The first user spent most of their time on infrastructure; the second spent most of their time on actual creative work.
Mirel's case studies show this isn't a minor distinction. In one scenario, 75% of an analyst's first-run analysis was "data conditioning" — purely enabling work. In the RCA (root cause analysis) team scenario, teammates spent more time fighting the programme's rigid task model than performing the actual analysis. In every case, the ratio of enabling-to-mainline work was a direct measure of how well the software supported the user's real goals.
How to Approach Classification
The key question for each tracked event: Does this action directly modify the creative output, or does it support the process of creating it?
| Mainline (Produces Output) | Enabling (Supports the Process) |
|---|---|
| Creating elements | Importing/converting assets |
| Editing properties | Browsing asset libraries |
| Composing/arranging | Configuring project settings |
| Animating | Organising layer hierarchy |
| Applying effects | Searching for tools/features |
| Rendering/exporting | Saving/loading workspace states |
| Drawing/painting | Comparing versions |
| Scripting creative logic | Annotating/commenting |
| Undoing to explore alternatives | |
| Adjusting preferences/settings |
This classification is inherently approximate — some actions straddle both categories (is organising layers preparation or composition?). But even rough classification provides valuable signal when tracked consistently over time.
What This Enables
With enabling-vs-mainline classification, you can surface insights that are invisible in standard analytics:
- Enabling task ratio by maturity: Do experts spend less time on enabling tasks? If not, the tool isn't reducing enabling burden with experience — a sign that the enabling tasks are structural overhead, not a learning cost.
- Enabling task ratio by session phase: Is enabling work front-loaded (first 10 minutes) or distributed throughout? Front-loaded enabling work suggests a "setup tax" on every session; distributed enabling work suggests the tool interrupts creative flow with housekeeping.
- Enabling task ratio trends: If the ratio increases across software versions, new features may be adding enabling burden faster than they add creative capability.
- Correlation with satisfaction: Sessions with high enabling ratios may correlate with lower satisfaction (if measured), even when feature-usage and session-duration metrics look healthy.
Connection to Other Analytics
Enabling task analytics complements two other metrics in this guide:
- Preparation Burden (in Workflow & Complexity) is a concrete, measurable subset — it specifically tracks preparatory-vs-creative action ratios within sessions. Think of it as the production-ready version of the enabling task concept, focused on the most instrumentable category of enabling work.
- Activity Profile Detection (earlier on this page) offers a different lens on the same phenomenon. The five CD activity types (search, transcription, incrementation, modification, exploratory design) classify how users work; enabling-vs-mainline classifies why — to produce output or to support the process of producing it. Used together, they reveal whether a user who is "searching" is doing so as part of creative exploration (mainline) or because they can't find something they need (enabling).
The practical path: start with Preparation Burden (measurable now), extend to Activity Profile Detection (requires more instrumentation), and aspire to full enabling task classification as your event taxonomy matures.