Free Text Analysis


Support Experience Transformation

Project Overview

As the design lead for a global employee support experience transformation initiative at EY, my focus was to eliminate barriers and challenges in traditional, technology-led support experiences. By conducting extensive user research and leveraging telemetry, Service Now, and survey data, we designed and delivered a new support experience optimized for ease of use, speed, and efficiency. The results yielded significant reductions in support costs, ticket volume, and improvements in employee productivity and satisfaction. The initiative also improved the agent experience by introducing automation and reducing repetitive requests, allowing them to focus on high-value interactions.
In support of the program, the UXD and UXR team engaged on a project to inform the information architecture of the Service Now Self-Service Portal Catalog. To ensure that our work was grounded in evidence and aligned with user mental models, my research partner, and I conducted a rigorous qualitative free text analysis. As a part of our analysis, we developed a replicable framework to deliver valuable insights to our analytics team, facilitating a deeper understanding of how users articulate their technology-related issues.

Methods and Limitations

Free text analysis is a valuable technique for designers and researchers to comprehend and classify different concepts within a given domain. It involves analyzing and organizing unstructured text data, providing insights into how people perceive and discuss information. In the realm of employee support experience transformation, where quantitative analyses may sometimes overlook nuanced insights, a qualitative free text analysis becomes crucial. Unlike broader quantitative approaches, this method allows for a deeper exploration of user sentiments and articulations related to technology issues. This qualitative check ensures that nuances are not lost, providing a more comprehensive understanding of user experiences.

In this analysis, we focused on the short description field in Service Now self-service ticket submissions, identifying common themes, topics, and patterns. To increase the effectiveness of this effort, we identified the top 10 ticket requests globally based on Service Now ticket volume data. For each request type, we collected and organized the free text extracted from the short description field, viewing between 30-50 tickets for each category in each country. In some countries, alternate categories were selected due to the absence of a specific catalog item.

It is important to note that this method is purely qualitative in nature and does not provide statistically significant insights. The goal of this analysis was to establish high-level catalog item naming guidelines and to provide a framework for the analytics team to extrapolate.

Process

To begin our analysis, we collected free text data from the top 10 catalog item form submissions with the highest monthly volume, and categorized by ticket category using distinct sticky note colors for each country.

A series of boards arranged by country with ticket category organized


We organized the colored-coded free text data stickies into corresponding ticket category boards, and performed a clustering analysis by identifying patterns in users' language usage, such as their use of descriptors like "broken," "not working," or "doesn't work" to describe malfunctions, or the verbs they use, such as "grant access" or "provide access." Additionally, we observed color concentration representing different countries. Based on this analysis, we were able to group the data into meaningful categories.

A series of boards arranged by country with ticket category organized


We then created a board that focused on the subject of each ticket category. Using this board, we began clustering naming variances and identifying commonalities. Additionally, by examining the variation and concentration of colors, we were able to determine whether a term was more globally or regionally used. For instance, we found that "laptop" was a globally common term when referring to computers, followed by "computer" and "PC" as the second and third most frequent terms, respectively. This helped us create aliases for improved search result accuracy. We utilized this information to recommend catalog item naming for top items and create a framework for supporting search result accuracy through the use of alias terms. This approach is highly flexible and can be customized to meet the specific needs of different information architecture projects.

A series of boards arranged by country with ticket category organized


Insights Summary


Intune Company Portal

Considering the prevalence of tickets received for both "Intune" and "Company Portal", we suggest combining the two names for better coverage of commonly used language, resulting in "Intune Company Portal".

Microsoft Teams

Although both "Teams" and "MS Teams" were used in the same number of countries, "Teams" was more commonly used by individuals. However, "MS Teams" is a strong keyword.

Microsoft Outlook

In most cases, people referred to the product as "Outlook" without including "MS" or "Microsoft." Similar to how people referred to "Teams," they also used product functions like "Email" and "Calendar" to describe issues. These words would be great as keywords and for bot training.

EY Remote Connect

There was a lot of variation in how people referred to EY's remote connection software, but "EY Remote Connect" is the most complete and accurate name. Most variations included a combination of "EY," "Remote," and "Connect." Because VPN usage is high in India, the USA, and China (all high-volume ticket countries), "VPN" should be an essential keyword.

RSA / SecurID

"RSA / SecurID" was not the most frequently used single term, but it was common across multiple countries and combines the two most common terms, "RSA token" and "SecurID," into one for better coverage. See the list of other phrases captured for how often "RSA" and "SecurID" were used in variations, and for keyword and bot training options. Other common terms include "Software Token," "RSA Hardware Token," "RSA Software Token," "SecurID Token," "SecurID VPN," "Token App," "Access Token," "RSA Security Token," "RSA Token Generator," "RSA Portal," "Security Token," "Secure Token," and "RSA code."

Mercury Timesheet

"Mercury Timesheet" was not the most frequently used single term, but it was common across multiple countries and combines the two most common terms, "Mercury" and "Timesheet," into one for better coverage.

EY AppStore

There were a few different variations in capitalization and spacing for "EY AppStore," which should be considered when creating keywords and for bot training:

  • EY app store
  • EY appstore
  • EY Appstore

EY Canvas

Although "EY Canvas" was not as widely used as just "Canvas," it includes "Canvas," so all issues should be covered by the proper name of "EY Canvas."

Phone

All references to "Phone" were related to installing Intune and other apps, so it is clear they were not talking about a desk phone.

Laptop

"Laptop" was overwhelmingly the most commonly used term. However, there were instances where people referred only to the part of the device causing the issue, such as "Battery," "Keyboard," "Camera," "Mouse," "Headphones," and "Charger."

  • Battery
  • Keyboard
  • Camera
  • Mouse
  • Headphones
  • Charger

Portfolio



Want to learn more about my experience and skills?
Download my resume and let's chat!

Download now