Get Started

Measuring CX Themes

Themes = What Makes and Breaks CX

Understanding the themes that drive customer experience enables CX teams to explain changes in customer sentiment and effort, and ultimately, business outcomes. It helps them achieve goals like improving outcomes, finding early warning signals to get ahead of customer expectations, and testing hypotheses about how to improve. 

Customers might adore your new UI but hate that it’s too hard to export reports. They might be blown away by how helpful your Support team is with order-related questions but think your delivery tracking functionality is a pain. There are infinite dynamics that shape what makes and breaks your customer experience — and these are themes. 

Themes vs. Anecdotes

According to Salesforce, 76% of customers expect companies to understand their needs. But 51% of customers say that most companies fall short of these expectations. So why is that?

Because too few CX teams actually measure themes. Most traditional CX metrics only reflect structured, quantitative data. Themes are patterns generated by the qualitative opinions and impressions your customers express organically using natural language across their journey, and they are company-specific. So there’s no universal metric like Net Promoter Score (“NPS”), Customer Satisfaction Score (“CSAT”), or Customer Effort Score (“CES”) that explicitly captures and measures themes. So instead, CX teams hunt for commonalities across the comments that customers sometimes leave accompanying survey ratings. 

But by definition, a theme is an idea that recurs. To meaningfully observe recurrence, you need to be analyzing a much higher volume of customer interactions at any given time — otherwise, you’re looking at anecdotes, not themes. 

When survey comments are sparse, a common instinct is to look to the volumes of organic unstructured feedback that customers leave every day in other channels like user forums, support cases, and so on. But there’s just too much there to capture and synthesize effectively with a manual approach. 

Measuring Themes is the Antidote to Inaction

Imagine you’re a utility company looking to reduce tree-related power outages. You ask your customers to call you when they see downed trees nearby — but you can’t depend on customers to report back. You could have teams patrolling overhead power lines to identify problem areas — but this is very expensive, both financially and logistically. So how do you get an accurate, scaled, cost-effective understanding of which parts of the power line are at risk at any given time? 

Fortunately, utility companies use an AI-enabled technology called LiDAR that models which trees share the structural and positional characteristics of fallen trees in the area. 

In this case, the reported problem trees are the anecdotes (or survey feedback) — and they often provide a solid starting point. But the utility company discovers themes by monitoring the physical characteristics of the trees day-in and day-out. 

Depending on surveys to illustrate themes also often means you miss an opportunity to discover the context in which they surface. Raise your hand if you’ve ever learned something interesting from a survey comment but then had to embark on a long and arduous fact-finding mission to understand why the customer said what they did? And how many other customers experienced that same thing?

Our utility companies monitor daily tree positioning so that they can understand the impact of themes like flooding, construction damage, and wind on tree-related power outages, to help predict and better prepare for them. They probably couldn’t count on a high enough volume of examples that neatly fit in each bucket if they only relied on reported problem trees. 

Themes aren’t useful unless you can analyze them on a relative basis. Raise your hand again if you’ve ever had deja vu in a cross-functional meeting where you’re repeatedly weighing the same set of potential CX priorities. But because you can’t point to underlying data and enough customer verbatims illustrating the context and impact, personality and politics win over customer-centric decisions. 

Without the ability to evaluate one theme’s impact over another, either the cross-functional stalemates continue, or you will chase one theme today and chase another one tomorrow. You’ll run hard in a lot of directions, but you won’t run very far. 

Measuring themes means you can have efficient cross-functional discussions and easily describe the impact of prioritizing any given decision over another. 

How Should Themes be Measured?

But measuring themes? How do you measure something qualitative? 

Natural Language Understanding technology (“NLU”) brings structure to your unstructured organic customer feedback by identifying which themes are moving the needle for which customers, when, and why. Across all of those “How do I…?”, “Is there an easier way to…?”, and “I wish I could…” statements are recurring themes where groups of customers are organically expressing similar causes of delight, issues, and deal-breakers. 

Organic customer feedback is a gold mine of customers expressing known themes but is also an excellent resource for surfacing new or unknown themes that haven’t appeared in survey comments. Regardless of whether themes are known or unknown, measuring organic customer feedback with NLU tracks the rate at which customers express these themes. By measuring organic customer feedback at scale, feedback intelligence tools can turn themes into their own entities or meta-data that you can track just like you would track certain customers or channels. They work similarly for CX teams the way that the LiDAR technology works for the utility company. 

But it doesn’t stop there. Some of the most common themes might be customers asking about something that would be nice-to-have but isn’t mission-critical. So you also need to measure how much negative or positive sentiment themes cause and how much customer effort and team effort you can attribute to them. 

It’s the utility company’s equivalent of knowing which trees are most likely to cause power outages. By modeling historical tree fall patterns against topography with AI, they can categorize the entire forest inventory by risk level and show exactly which trees have the highest risk of falling. Measuring themes to identify what drives customer sentiment and customer effort is how you figure out which themes have the most significant impact on your customer experience. 

Then, the question becomes, how do you test hypotheses about the most effective ways to address those themes? 

Imagine you work for a health insurance company. There’s an ongoing debate about potentially launching a new plan type or merging two existing options — let’s call them plan A and plan B. Plan A is more expensive for customers and costs your business more to support, but it offers more coverage on lab tests than plan B. 

You’re seeing customers downgrade to plan B, and an uptick in Support interactions with plan A customers, specifically for dermatology coverage — and you need to know why. The question is, do customers care enough about the additional lab test coverage that Plan A offers to pay more for it? 

Your Provider team says yes. They worry that customers downgrading to plan B will make lab tests too expensive to support customers who need them. Your Consumer team says no. They believe that merging the plans will reduce customer effort and cost of service enough to make up for any additional costs for lab testing. 

So you analyze the last month of interactions related to dermatology coverage with plan A and plan B customers to understand the real impact of the differences between the two plans on your customer experience. 

NEW summary tableYour takeaway: Plan A customers are upset because they think they paid more for greater flexibility to see specialists, but plan A’s specialist coverage is the same as plan B’s. Unfortunately, it’s not clear that plan A’s flexibility and higher price tag are more about lab testing than specialist visits. However, your Provider team makes a valid point that plan A customers value their lab coverage, and you don’t want to rock the boat unnecessarily. 

measuring themes - verbatim example

In summary, using organic customer feedback, you’ve identified that limited specialist coverage drives negative sentiment and high customer effort across your plan A customers. By reviewing the verbatims, you get a strong sense of how this unfolds in context. Next, you can quantify the impact and set goals to improve sentiment and reduce effort for plan A customers. 

Both your Provider and your Consumer team can now clearly see the complete picture. Your organic customer feedback and the themes you measured directly from it have served as a Rosetta Stone to bring different functions onto the same page. You’ve defined a shared problem, and they both have the information that they need to test hypotheses about how to improve. 

For example, your Consumer team might introduce a quiz for new customers, where they can proactively recommend the right plan based on their preferences and medical conditions. You might also grant your plan A customers more flexibility to see specialists in exchange for conducting lab tests when necessary only via your Provider team’s preferred partner. 

Goal Increase sentiment and reduce effort for plan A customers

How do you know it’s working? Gradually, you should see a reduction in plan A customer effort associated with inquiries about seeing a specialist, and at the same time, an increase in customer sentiment. 

Instead of introducing a new plan type and potentially causing more customer confusion, you have pinpointed the themes driving the customer behaviors you want to encourage and avoid. 

Back to the utility company — by focusing on removing trees in high-risk areas to reduce power outages, they spend less budget in areas with significantly fewer trees at risk. Now you, too, are focusing your effort on a targeted, data-driven problem you can quantify and then measure the results of your efforts. 

Measuring themes, not anecdotes, from organic customer feedback, is how you figure out which ones have the greatest impact on your customer experience and where the consequences of inaction will be most meaningful. And how CX teams increase customer sentiment, reduce customer effort, and lower their cost of service.

Mary Cleary

Written by Mary Cleary