Skip to main content

Usability Analytics for Complex Software

Consumer analytics will actively mislead you when applied to complex professional software. DAU/MAU ratios, session length, conversion funnels, A/B test click rates — none of them measure what matters. A 3-hour session isn't "more engaged" than a 20-minute one if the user spent 2.5 hours fighting the interface. High daily active users doesn't mean your software is well-designed if usage is organisationally mandated. Standard A/B testing optimises for clicks, not for utility.

Complex software needs fundamentally different measurement. Everything in this section is designed for software where the goal is enabling deep, expert-level work — not maximising time-on-site or goosing conversion funnels.

What to Measure Instead​

The right questions for complex software look nothing like consumer analytics:

  • Are users becoming more sophisticated over time, or plateauing at a basic level?

  • Which features are users discovering, and which remain invisible?

  • Where in the learning curve do users give up?

  • Are users building increasingly complex work, or sticking to simple projects?

  • What friction patterns predict churn before it happens?

  • Are expert users finding efficient workflows, or stuck on verbose ones?

  • Which Usability Harmonics dimensions are currently problematic, and are interventions working?

  • Where do users spend their time navigating vs. working?

  • How much time goes into preparing to work vs. actually working?

Answering these requires different instrumentation, different visualisations, and different interpretive frameworks than anything in the consumer analytics playbook.

The UX Dashboard​

These analytics should feed a living dashboard that product teams review regularly — not a one-off study conducted when someone suspects a problem. The Conceptual Burden section of Usability Harmonics already describes how conceptual gradient and concept barrier visualisations make excellent additions to a UX dashboard. The analytics here extend that idea into a comprehensive measurement practice.

The dashboard also serves as a pre-evaluation triage tool for Usability Harmonics evaluations. The Analytics for Usability Evaluation page describes how specific analytics patterns can focus evaluator attention on the dimensions most likely to have actionable problems — directing expert attention where it matters most.

The Paradox of the Active User​

A recurring theme across these analytics is the Paradox of the Active User — Carroll and Rosson's (1987) finding that users' performance "reaches an asymptote at a level of mediocrity" despite years of daily use. Users reuse known methods rather than discovering better ones — what the research calls "production bias."1 This paradox surfaces in sophistication plateaus, method stagnation, feature non-discovery, and many other signals throughout this section. The core insight: experience alone does not produce expertise — users need design support to progress.

Never Rely on a Single Metric

Research consistently shows that effectiveness, efficiency, and satisfaction don't always agree: "a system may be effective and efficient to use, but users may hate it. Or the other way round"2. Professional users can be highly productive with systems they dislike — creating false positives in productivity-only metrics. Always measure across multiple dimensions.

How This Section is Organised​

Each analytics type in the following pages follows a consistent structure:

  • What it measures — the metric and what data it requires

  • Why it matters — why this is specifically important for complex professional software (not just generic "engagement")

  • How to implement — practical guidance on instrumentation, with real-world examples

  • What to watch for — how to interpret the data and what patterns indicate problems

The analytics are grouped into five areas:

  1. Understanding Your Users — Who are your users, how sophisticated are they, and how is that changing?
  2. Learning & Expertise — Are users progressing along the learning curve, and where do they get stuck?
  3. Retention & Churn — Where and why do users leave, and what predicts it?
  4. Workflow & Complexity — What are users actually building, and how efficiently?
  5. Analytics for Usability Evaluation — How analytics support the Usability Harmonics testing process, including pre-evaluation triage, post-intervention validation, and continuous monitoring.

Footnotes​

  1. Carroll, J. M., & Rosson, M. B. (1987). Paradox of the active user. In J. M. Carroll (Ed.), Interfacing Thought: Cognitive Aspects of Human–Computer Interaction (pp. 80–111). MIT Press. ↩

  2. Wachowicz, J. (2007). Surveys – Questionnaires. In COST294-MAUSE 3rd International Workshop: Review, Report and Refine Usability Evaluation Methods, Athens, pp. 20–21. ↩