Project Overview

As the design lead for a global employee support experience transformation initiative at EY, my focus was to eliminate barriers and challenges in traditional, technology-led support experiences, impacting over 400k employees worldwide. By conducting extensive user research and leveraging telemetry, Service Now, and survey data, we designed and delivered a new support experience optimized for ease of use, speed, and efficiency. The results yielded significant reductions in support costs, ticket volume, and improvements in employee productivity and satisfaction. The initiative also improved the agent experience by introducing automation and reducing repetitive requests, allowing them to focus on high-value interactions.

As a part of this program, I collaborated with a partner from UXR to inform the information architecture of the Service Now Self-Service Portal Catalog. It was crucial for us to ensure that the language and categorization used in the portal resonated with users' mental models, making it easier for them to find the support they need. To ensure our approach was evidence-based and aligned with user expectations, my research partner and I conducted a rigorous qualitative free text analysis. This analysis aimed to capture how users articulate their technology-related issues in their own words, providing valuable insights into their needs and preferences. As a result, we developed a replicable framework to deliver actionable insights to our analytics team, facilitating a deeper understanding of user behavior and language patterns.

Methods and Limitations

Free text analysis is a valuable technique for designers and researchers to comprehend and classify different concepts within a given domain. It involves analyzing and organizing unstructured text data, providing insights into how people perceive and discuss information. In the realm of employee support experience transformation, where quantitative analyses may sometimes overlook nuanced insights, a qualitative free text analysis becomes crucial. Unlike broader quantitative approaches, this method allows for a deeper exploration of user sentiments and articulations related to technology issues. This qualitative check ensures that nuances are not lost, providing a more comprehensive understanding of user experiences.

In this analysis, we focused on the short description field in Service Now self-service ticket submissions, identifying common themes, topics, and patterns. To increase the effectiveness of this effort, we identified the top 10 ticket requests globally based on Service Now ticket volume data. For each request type, we collected and organized the free text extracted from the short description field, viewing between 30-50 tickets for each category in each country. In some countries, alternate categories were selected due to the absence of a specific catalog item.

It is important to note that this method is purely qualitative in nature and does not provide statistically significant insights. The goal of this analysis was to establish high-level catalog item naming guidelines and to provide a framework for the analytics team to extrapolate.

Process

To begin our analysis, we collected free text data from the top 10 catalog item form submissions with the highest monthly volume, and categorized by ticket category using distinct sticky note colors for each country.

We organized the colored-coded free text data stickies into corresponding ticket category boards, and performed a clustering analysis by identifying patterns in users' language usage, such as their use of descriptors like "broken," "not working," or "doesn't work" to describe malfunctions, or the verbs they use, such as "grant access" or "provide access." Additionally, we observed color concentration representing different countries. Based on this analysis, we were able to group the data into meaningful categories.

We then created a board that focused on the subject of each ticket category. Using this board, we began clustering naming variances and identifying commonalities. Additionally, by examining the variation and concentration of colors, we were able to determine whether a term was more globally or regionally used. For instance, we found that "laptop" was a globally common term when referring to computers, followed by "computer" and "PC" as the second and third most frequent terms, respectively. This helped us create aliases for improved search result accuracy. We utilized this information to recommend catalog item naming for top items and create a framework for supporting search result accuracy through the use of alias terms. This approach is highly flexible and can be customized to meet the specific needs of different information architecture projects.

Insights Summary

"Intune Company Portal"

Considering the prevalence of tickets received for both "Intune" and "Company Portal", we suggest combining the two names for better coverage of commonly used language, resulting in "Intune Company Portal".

"Microsoft Teams"

Although both "Teams" and "MS Teams" were used in the same number of countries, "Teams" was more commonly used by individuals. However, "MS Teams" is a strong keyword.

"Microsoft Outlook"

In most cases, people referred to the product as "Outlook" without including "MS" or "Microsoft." Similar to how people referred to "Teams," they also used product functions like "Email" and "Calendar" to describe issues. These words would be great as keywords and for bot training.

"EY Remote Connect"

There was a lot of variation in how people referred to EY's remote connection software, but "EY Remote Connect" is the most complete and accurate name. Most variations included a combination of "EY," "Remote," and "Connect." Because VPN usage is high in India, the USA, and China (all high-volume ticket countries), "VPN" should be an essential keyword.

"RSA / SecurID"

"RSA / SecurID" was not the most frequently used single term, but it was common across multiple countries and combines the two most common terms, "RSA token" and "SecurID," into one for better coverage. See the list of other phrases captured for how often "RSA" and "SecurID" were used in variations, and for keyword and bot training options. Other common terms include "Software Token," "RSA Hardware Token," "RSA Software Token," "SecurID Token," "SecurID VPN," "Token App," "Access Token," "RSA Security Token," "RSA Token Generator," "RSA Portal," "Security Token," "Secure Token," and "RSA code."

"Mercury Timesheet"

"Mercury Timesheet" was not the most frequently used single term, but it was common across multiple countries and combines the two most common terms, "Mercury" and "Timesheet," into one for better coverage.

"EY AppStore"

There were a few different variations in capitalization and spacing for "EY AppStore," which should be considered when creating keywords and for bot training:

• EY app store
• EY appstore
• EY Appstore

"EY Canvas"

Although "EY Canvas" was not as widely used as just "Canvas," it includes "Canvas," so all issues should be covered by the proper name of "EY Canvas."

"Phone"

All references to "Phone" were related to installing Intune and other apps, so it is clear they were not talking about a desk phone.

"Laptop"

"Laptop" was overwhelmingly the most commonly used term. However, there were instances where people referred only to the part of the device causing the issue, such as "Battery," "Keyboard," "Camera," "Mouse," "Headphones," and "Charger."

• Battery
• Keyboard
• Camera
• Mouse
• Headphones
• Charger

Reflection: Human-led Free Text Analysis Empowered by AI

The ongoing advancements in AI present a compelling opportunity to enhance and expand the capabilities of a human-led free text analysis within the realm of user experience design research. While the qualitative nature of free text analysis can prove instrumental in capturing nuanced insights and understanding user sentiments, integrating AI-powered tools can expand this process by offering scalability and efficiency.

Human-Led Framework for NLP Integration

Human-led free text analysis establishes a structured framework by identifying key themes, sentiments, and patterns in unstructured text data, accounting for nuances and context that AI may overlook independently. The human-guided framework serves as a foundational dataset that AI-driven NLP algorithms can integrate and expand upon, enabling faster extraction of insights and greater understanding of user interactions with nuanced contexts.

Guiding Sentiment Analysis with Human Insights

Human-led analysis provides valuable insights into user emotions, attitudes, and contextual nuances within free text data, ensuring that AI-driven sentiment analysis accounts for complex emotions and subtle variations in sentiment expression. AI models for sentiment analysis can leverage this human-led framework to accurately identify sentiment patterns, refine sentiment categorizations, and adapt to evolving user sentiments over time while considering nuanced contexts.

Framework-Based Topic Modeling for AI Enhancements

The structured framework established through human-led topic identification and clustering accounts for nuances in nomenclature in topic categorization, ensuring that AI-powered topic modeling algorithms accurately capture and categorize diverse topics within free text data. By integrating human-derived topic categories and trends, AI can automate categorization, identify emerging topics, and enhance the accuracy of topic clustering within nuanced contexts.

Importance of Human Touch

While AI offers significant advancements in data processing and analysis, it's critical to recognize the importance of the human touch in interpreting and contextualizing findings. The integration of AI technologies holds immense potential to amplify the capabilities of qualitative exercises such as free text analyses, paving the way for more scalable, efficient, and insightful approaches to understanding user experiences.