Analytics for Usability Evaluation
Usability Harmonics provides a structured way to evaluate complex software, but evaluations are periodic and manual. Between them, how do you know whether problems are emerging, interventions are working, or a new release has disrupted an established workflow? Analytics provide continuous, automated signals between manual evaluations.
Analytics can't replace manual evaluation. They measure behaviour, not cognition — a high undo rate may reflect difficulty, experimentation or comparison. But they can:
- Prioritise which dimensions need attention before you run a full evaluation
- Validate whether design interventions had the intended effect
- Detect regressions between evaluations, especially after releases
What follows describes how to use the analytics from the preceding sections as a companion to Usability Harmonics testing.
Understanding Experience Over Time
Usage patterns can show where behaviour changes. To understand how users experience those changes, ask established users to sketch their relationship with the product over time and explain its turning points. Discovery, recurring frustrations and changes in working circumstances may all help explain a curve that the dashboard alone cannot.
Pre-Evaluation Triage
Before running a full Usability Harmonics evaluation, review your analytics dashboard to identify which dimensions are most likely to have problems. Hvannberg and colleagues compared heuristic sets and reporting methods in an evaluation of a student portal. Its 2 × 2 experiment compared two heuristic sets and paper versus tool-based reporting, using 19 final-year computing students and one graduate. The 21% overall effectiveness was the product of validity (predicted problems confirmed in user testing) and thoroughness (user-test problems predicted). The 0.046 any-two agreement excluded one evaluator who reported no problems; the figure for all 20 was 0.043. They are not universal rates, and the study does not establish that expertise matters more than method.2 Use analytics to prepare questions, while leaving room for problems the dashboard cannot see.
| If analytics show... | Investigate this dimension | Using this analytics type |
|---|---|---|
| Users plateau at intermediate sophistication levels | Conceptual Burden; confirm an unmet task need | Sophistication Scoring, Skill Progression Curves |
| Low adoption for features at a specific tier | Visibility; check need and opportunity | Feature Adoption Funnel, Feature Discovery Rate |
| High undo/error rates that persist across maturity levels | Error-Proneness; distinguish errors from exploration | Friction Signals |
| Veterans use the same creation methods as novices | Visibility, Premature Commitment | Creation Method Distribution, Performance Dip Tracking |
| Novice and expert workflow patterns differ radically | Diffuseness, Rigidity | Workflow Pattern Analysis |
| Users produce complex work but churn anyway | Hard Mental Operations, Rigidity | Project Complexity + Churner Segmentation |
| "Struggled" churner segment dominates | Conceptual Burden; ask why use stopped | Churner Segmentation, First Session Quality |
| Regression clusters after software updates | Consistency | Plateau and Regression Detection |
| Slow Starters never accelerate | Conceptual Burden; confirm an unmet need | New User Onboarding Journeys |
| Wide P25-P75 bands in skill progression | Consistency; check task and cohort differences | Skill Progression Curves |
If your analytics point strongly to two or three dimensions, brief your evaluators to focus there first. Blackwell & Green's Cognitive Dimensions research shows that activity profiles determine which dimensions matter most — analytics serve as a proxy for identifying the dominant activity profiles before evaluation begins.
Post-Evaluation Validation
After making design changes based on a Usability Harmonics evaluation, specific analytics can assess the effect. This connects qualitative evaluation with quantitative measurement.
Conceptual Burden Intervention
You have redesigned a feature to reduce conceptual weight, perhaps by reducing the number of concepts required for a workflow or improving the conceptual gradient by reordering the learning path.
Track:
- Sophistication score distribution: Does it shift upward?
- Plateau band: Does the Intermediate plateau thin?
- Skill progression curves: Does the median curve steepen at the intervention point?
- Session milestone funnel: Does the drop-off at the targeted milestone decrease?
Observation period: choose a period with enough comparable task opportunities and practice. Four to eight weeks may be a planning example, but is not an established evaluation window. Check seasonal variation and changes in cohort composition.
Visibility Intervention
You have made a hidden feature more discoverable by presenting it in context, adding progressive disclosure cues or restructuring the interface hierarchy.
Track:
- Feature adoption rate: Does it increase for the target feature?
- Feature discovery session: Does the median session number decrease? (Users find it sooner.)
- Feature adoption funnel: Does the drop-off at the intervention point decrease?
Observation period: follow enough users with a relevant need and opportunity to discover the feature. Two to four weeks is only a planning example.
Error-Proneness Intervention
You have redesigned a workflow to reduce errors by adding constraints, improving feedback or restructuring the interaction sequence.
Track:
- Undo rate for the specific workflow: Does it decrease?
- Error encounter rate for the target error type: Does it decrease?
- Undo rate segmented by maturity: Does the error rate decrease for both new and veteran users? If it only drops for new users, the fix may not address the veteran workflow.
Observation period: collect enough attempts at the affected task to compare errors, output quality and recovery. A fixed two- to four-week window may not suit infrequent work.
Rigidity Intervention
You have reduced the cost of changing approach through better undo/redo support, non-destructive editing or easier switching between methods.
Track:
- Creation method distribution: Does it shift? (More users adopting the more flexible approach.)
- Performance dip reversion rate: Does it decrease? (Users persist through the transition more often.)
- Workflow pattern analysis: Do expert and novice patterns converge? (The efficient path becomes accessible earlier.)
Observation period: allow relevant practice and a later task. Four to eight weeks is illustrative; check whether the new method remains useful after the immediate transition.
Continuous Monitoring
Review a small set of sentinel metrics each week, without waiting for the next evaluation. These provide early warning of dimension regressions without requiring a full dashboard review.
Sentinel Metrics
The thresholds below are illustrative alert settings, not validated benchmarks. Calibrate against task, role, opportunity, normal variation and instrumentation changes. An alert requests investigation; it does not diagnose a dimension.
| Sentinel | What to Watch | Dimension | Alert Threshold |
|---|---|---|---|
| Sophistication shift | Median sophistication score across active users | Conceptual Burden | Drops 2+ points over 4 weeks |
| Discovery rate | Median discovery session for recently launched features | Visibility | Above 20 sessions for a feature intended for intermediate users |
| Veteran friction | Undo/error rate for users with 50+ sessions | Error-Proneness | Increases 15%+ over 4 weeks |
| Method stagnation | % of Regular/Veteran users still using novice methods | Premature Commitment | Above 60% for methods where an efficient alternative exists |
| Regression count | Users whose sophistication dropped 3+ points this period | Consistency | More than 5% of active users, especially after a release |
Reading the Sentinels
Sentinels should allow you to assess product health in under a minute. Look for the following patterns:
- All sentinels stable: no change detected by these measures. Continue the planned evaluation; unmeasured difficulties may remain.
- One sentinel triggered: Investigate that specific dimension before the next evaluation. Likely a localised issue.
- Multiple sentinels triggered simultaneously: check whether something systemic has changed — a major release, a competitor shift, a platform change. Consider an ad-hoc evaluation focused on the affected dimensions.
- Sentinel triggered immediately after a release: High priority. The release may have introduced a regression. Cross-reference with Plateau and Regression Detection to identify affected users and features.
Aspirational: Activity Profile Detection
This proposed analytics approach requires validation, as do the interpretations and thresholds above. It requires more instrumentation effort and draws from the theoretical framework rather than production-validated implementations.
The Cognitive Dimensions framework distinguishes activities such as transcription, modification and exploratory design.3 Their requirements differ, so aggregate metrics can hide important differences in work.
An undo may reverse an error or support a deliberate experiment in any of these activities. Activity context narrows the questions to ask; it does not by itself determine whether an action was useful.
The Five Activity Types
The Cognitive Dimensions framework defines five core activities when working with notational systems:
| Activity | Behavioural Signals | Key UH Dimensions |
|---|---|---|
| Search | High navigation-to-action ratio, frequent view switches, minimal creation events | Visibility, Hard Mental Operations |
| Transcription | Steady creation rate, low undo, low exploration, linear progression | Consistency, Diffuseness |
| Incrementation | Small modifications to existing work, low creation rate, frequent property changes | Rigidity (viscosity), Error-Proneness |
| Modification | High selection-change rate, property edits on existing elements, moderate undo | Rigidity, Invisible Links |
| Exploratory Design | High creation-and-deletion rate, diverse feature usage, non-linear workflow, frequent undo | Premature Commitment, Provisionality |
How This Could Be Implemented
Start with observed and annotated task segments, then test whether event patterns can distinguish the relevant activities. Five-minute windows are an illustrative choice, not an established unit. Report classification errors and ambiguous segments; an unreliable label may make interpretation worse.
With activity detection you could produce:
-
Friction Signals segmented by activity: An illustrative comparison is 12 undos/hour during exploration and 8 during transcription. Neither rate establishes success or failure; inspect the reasons and consequences.
-
Feature discovery by activity: "Users in exploratory mode discover 3x more features per session than users in transcription mode — but only 15% of sessions are exploratory."
-
Time allocation across activities: "Expert users spend 40% of time in modification, 25% in exploration. Novice users spend 60% in search, 20% in transcription." These illustrative proportions require observed activity labels and a reliable measure of time. Search may be part of the work, rather than avoidable navigation.
What This Enables
Classifying analytics by activity makes the Pre-Evaluation Triage table more specific. Instead of "high undo rate → investigate Error-Proneness", you get "high undo rate during transcription → investigate Error-Proneness for transcription workflows" — a much more targeted investigation.
Aspirational: Wayfinding Analytics
This analytics approach addresses a gap identified in Mirel's research on complex problem solving. It requires navigation-specific instrumentation that most applications don't yet provide.
Barbara Mirel's (2004) research on complex problem solving documented that wayfinding — navigating information landscapes, re-finding previous states, maintaining orientation — is the least designed-for activity in professional software, yet it consumes enormous time and cognitive effort. Users spend substantial portions of their sessions on what Mirel calls "enabling tasks" — work that supports the work but doesn't directly produce output. Wayfinding is one category of enabling task; the full scope is covered in the Enabling Task Analytics section below.
Compare measured time and task consequences before prioritising an intervention. Reducing 40 minutes of navigation by 10% saves four minutes; reducing 60 minutes of other work by 10% saves six. The greater opportunity depends on both the burden and the feasible improvement.
What to Instrument
| Signal | What It Reveals | UH Dimension |
|---|---|---|
| View/panel switches per minute | Navigation intensity — high rates suggest users can't find what they need | Visibility |
| Search-to-find ratio | How many searches it takes to find a specific item | Visibility, Hard Mental Operations |
| Repeated searches for the same item | Persistent findability problems | Visibility |
| Bookmark creation and return rates | Whether users build and use their own navigation shortcuts | Secondary Notation |
| Undo-as-navigation | Using undo not to fix errors but to return to a previous state | Provisionality, Premature Commitment |
| Navigation ratio | Time spent navigating (view switches, searching, scrolling without acting) vs. time spent acting (creating, modifying, configuring) | Overall wayfinding health |
What Healthy Patterns Look Like
-
Navigation ratio by familiarity: compare similar tasks. Continued navigation may reflect a more complex task or deliberate comparison, as well as a findability problem.
-
Low repeated-search rate: Users find things on the first or second try. High repeat rates suggest search doesn't match user mental models.
-
Bookmark usage correlates with productivity: Check task quality and avoidable effort, not project size alone. If bookmark usage is very low, the bookmarking system may be too hidden or too limited.
-
Undo-as-navigation is rare: Users have proper navigation tools — history, state management, bookmarks — and do not need to rely on undo to revisit previous states. High rates may warrant better comparison tools, but inspect whether the existing history is serving the intended task.
Connection to the Four Micro Contextual Questions
Wayfinding signals can help investigate the four micro contextual questions. Corroborate each interpretation through observation:
- "What have I done up to this point?" — High undo-as-navigation suggests users can't review their history effectively.
- "What did I just do?" — Repeated identical actions suggest the software isn't providing clear feedback on what just happened.
- "What's the current state?" — High view-switching rates suggest users can't assess current state from their current view.
- "What can be done next?" — High search rates for operations suggest users can't see what actions are available in context.
Aspirational: Enabling Task Analytics
This analytics framework addresses a fundamental insight from Mirel's research that current analytics approaches largely ignore. It requires a conceptual shift in how events are classified, not just new instrumentation.
Barbara Mirel's (2004) research draws a crucial distinction between two types of work in complex software:
- Mainline tasks: Core operations that directly produce output — creating elements, editing properties, composing, animating, rendering, exporting. This is what users think of as "the work."
- Enabling tasks: Work that supports mainline tasks but doesn't directly modify the creative output — annotating, comparing versions, bookmarking positions, capturing rationale, organising workspaces, searching for assets, configuring settings, preparing data. These tasks can be essential to interpreting the work and coordinating decisions.
Why Current Analytics Miss This
Standard feature-usage metrics count all actions equally. A session with 80% enabling tasks and 20% mainline tasks looks identical to a session with 20% enabling and 80% mainline — same total action count, same session duration, same "engagement." But these are completely different user experiences. The event proportions differ, but do not establish time proportions. Measure duration separately, account for overlapping activity, and distinguish necessary preparation from avoidable effort.
Mirel's case studies show the importance of this distinction. The case accounts include substantial data conditioning before analysis can proceed. In the RCA (root cause analysis) team scenario, team members spent more time working around the software's rigid task model than performing the analysis. These are contextual accounts, not validation of a universal ratio. Enabling work may improve quality, comparison or handoff; a lower ratio is not automatically better.
How to Approach Classification
For each tracked event, ask: Does this action directly modify the creative output, or does it support the process of creating it?
| Mainline (Produces Output) | Enabling (Supports the Process) |
|---|---|
| Creating elements | Importing/converting assets |
| Editing properties | Browsing asset libraries |
| Composing/arranging | Configuring project settings |
| Animating | Organising layer hierarchy |
| Applying effects | Searching for tools/features |
| Rendering/exporting | Saving/loading workspace states |
| Drawing/painting | Comparing versions |
| Scripting creative logic | Annotating/commenting |
| Undoing to explore alternatives | |
| Adjusting preferences/settings |
This classification is inherently approximate — some actions straddle both categories (is organising layers preparation or composition?). But even rough classification provides valuable signal when tracked consistently over time.
What This Enables
With enabling-vs-mainline classification, you can identify patterns that standard analytics cannot reveal:
- Enabling task ratio by maturity: Do experts spend less time on enabling tasks? Compare similar tasks and investigate the purpose of the enabling work before calling it overhead.
- Enabling task ratio by session phase: Is enabling work front-loaded (first 10 minutes) or distributed throughout? Front-loaded enabling work suggests a "setup tax" on every session; distributed enabling work suggests the tool interrupts creative flow with housekeeping.
- Enabling task ratio trends: If the ratio increases across software versions, new features may be adding enabling burden faster than they add creative capability.
- Correlation with satisfaction: Sessions with high enabling ratios may correlate with lower satisfaction (if measured), even when feature-usage and session-duration metrics look healthy.
Connection to Other Analytics
Enabling task analytics complements two other metrics in this guide:
- Preparation Burden (in Workflow & Complexity) is a concrete, measurable subset — it specifically tracks preparatory-vs-creative action ratios within sessions. It provides a proposed starting point for enabling-task measurement, focused on the category of work most readily instrumented.
- Activity Profile Detection (earlier on this page) offers a different lens on the same phenomenon. The five CD activity types (search, transcription, incrementation, modification, exploratory design) classify how users work; enabling-vs-mainline classifies why — to produce output or to support the process of producing it. Used together, they reveal whether a user who is "searching" is doing so as part of creative exploration (mainline) or because they can't find something they need (enabling).
Start with Preparation Burden, which is measurable now, and extend to Activity Profile Detection as you add the necessary instrumentation. Work towards full enabling task classification as your event taxonomy develops.