Usability Analytics for Complex Software

Consumer analytics can mislead when their interpretation is carried into complex professional software unchanged. DAU/MAU ratios, session length, conversion funnels and click rates describe activity, but they do not establish whether the work is going well. A 3-hour session isn't "more engaged" than a 20-minute one if the user spent 2.5 hours struggling with the interface. A high daily active user count does not mean your software is well designed if the organisation requires people to use it. An A/B test is only as useful as its outcome measure. Compare task quality, successful completion and avoidable effort when those are the questions at issue.

Complex software requires a fundamentally different approach to measurement. The analytics in this section are designed for software whose purpose is enabling deep, expert-level work, rather than maximising time-on-site or increasing conversion rates.

Interpreting the examples

The scores, thresholds, timelines and sample figures in this section are proposed starting points, not validated benchmarks. Calibrate them against your users' roles, tasks and working conditions. Telemetry records behaviour; observation and discussion help establish its meaning.

What to Measure Instead

Complex software requires different questions from those asked in consumer analytics:

• Are users becoming more sophisticated over time, or plateauing at a basic level?

• Which features are users discovering, and which remain invisible?

• Where in the learning curve do users give up?

• Can users produce the work their briefs require, including demanding work when it is needed?

• What friction patterns predict churn before it happens?

• Are expert users finding efficient workflows, or stuck on verbose ones?

• Which Usability Harmonics dimensions are currently problematic, and are interventions working?

• Where do users spend their time navigating vs. working?

• How much time goes into preparing to work vs. actually working?

Answering these questions requires instrumentation, visualisations and interpretive frameworks quite different from those used in consumer analytics.

The UX Dashboard

These analytics should inform a regularly updated dashboard that product teams review routinely. A single study conducted when someone suspects a problem is not sufficient. The Conceptual Burden section of Usability Harmonics already describes how conceptual gradient and concept barrier visualisations contribute to a UX dashboard. The analytics here extend that approach into a comprehensive measurement practice.

The dashboard also serves as a pre-evaluation triage tool for Usability Harmonics evaluations. The Analytics for Usability Evaluation page describes how specific analytics patterns can direct evaluators towards the dimensions most likely to have problems they can address.

The Paradox of the Active User

A recurring theme across these analytics is the Paradox of the Active User — Carroll and Rosson's (1987) finding that users' performance "reaches an asymptote at a level of mediocrity" despite years of daily use. Users reuse known methods rather than discovering better ones — what the research calls "production bias."¹ This paradox surfaces in sophistication plateaus, method stagnation, feature non-discovery, and many other signals throughout this section. The central point is that experience alone does not produce expertise. Users need design support to progress.

Never Rely on a Single Metric

Research consistently shows that effectiveness, efficiency, and satisfaction don't always agree: "a system may be effective and efficient to use, but users may hate it. Or the other way round"². Professional users can be highly productive with systems they dislike — creating false positives in productivity-only metrics. Always measure across multiple dimensions.

Evaluate What Users Gain as Well as What They Produce

Evaluation should consider what using a creative tool does for the person as well as the work they produce. Combine performance measures with evidence of users’ developing understanding and ability to work independently. Claims about lasting benefits require observation over time.

Research reviewC&C 2025

Beyond Productivity

Cox and colleagues reviewed 173 papers evaluating creativity support tools, published in the ACM Digital Library between 2015 and 2024. They examined the outcomes researchers measured, grouping them into user experience, creative output quality and benefits to users themselves, such as learning, confidence or well-being.³

Roughly 90% assessed user experience and 54% assessed output quality, whilst only 15% assessed benefits to users themselves. The review identifies a gap in what evaluations measure; it does not establish that the tools provided no such benefits.

PDF

How This Section is Organised

Each analytics type in the following pages follows a consistent structure:

• What it measures — the metric and what data it requires

• Why it matters — why this is specifically important for complex professional software (not just generic "engagement")

• How to implement — practical guidance on instrumentation, with real-world examples

• What to watch for — how to interpret the data and what patterns indicate problems

The analytics are grouped into five areas:

1. Understanding Your Users — Who are your users, how sophisticated are they, and how is that changing?
2. Learning & Expertise — Are users progressing along the learning curve, and where do they get stuck?
3. Retention & Churn — Where and why do users leave, and what predicts it?
4. Workflow & Complexity — What are users actually building, and how efficiently?
5. Analytics for Usability Evaluation — How analytics support the Usability Harmonics testing process, including pre-evaluation triage, post-intervention validation, and continuous monitoring.

Check the Interpretation

Check	Practical question or example
Design decision	What does the event measure, and which explanation would challenge your conclusion?
Failure to look for	A rising undo rate is labelled a defect even though users are comparing successful alternatives.
Verification task	Observe comparable tasks, annotate why undo occurred and compare output quality and independence. Validate any threshold on separate data before relying on it.

Footnotes

1. Carroll, J. M., & Rosson, M. B. (1987). Paradox of the active user. In J. M. Carroll (Ed.), Interfacing Thought: Cognitive Aspects of Human–Computer Interaction (pp. 80–111). MIT Press. ↩

2. Wachowicz, J. (2007). Surveys – Questionnaires. In COST294-MAUSE 3rd International Workshop: Review, Report and Refine Usability Evaluation Methods, Athens, pp. 20–21. ↩

3. Cox, S. R., Djernæs, H. B., & van Berkel, N. (2025). Beyond Productivity: Rethinking the Impact of Creativity Support Tools. Proceedings of C&C 2025. Sections 3–5. Published paper. ↩