🤖 Generative AI boosts productivity and performance of customer support, study finds

A new study by researchers from Stanford University and the National Bureau of Economic Research (NBER) has found that a generative AI-based conversational assistant can improve the productivity and performance of customer support agents.

The study, titled “Generative AI at Work”, used data from 5,179 agents who work for a large online retailer and analyzed how their behavior and outcomes changed after they gained access to the tool.

The tool, which was introduced in a staggered fashion across different teams and locations, uses natural language processing and deep learning to generate responses to customer inquiries based on previous interactions and best practices.

How the AI assistant works

The AI assistant is a generative model that can produce natural language responses to customer inquiries. The model is trained on a large corpus of historical conversations between customers and agents, as well as a set of guidelines and templates for best practices. The model can also learn from feedback and ratings from customers and agents.

The AI assistant operates in two modes: suggestion and automation. In suggestion mode, the AI assistant proposes a response to the agent, who can either accept it, edit it, or reject it. In automation mode, the AI assistant directly sends the response to the customer, without requiring the agent’s approval. The agents can switch between the two modes depending on their preferences and confidence.

How the AI assistant improves productivity and performance

The researchers found that access to the AI assistant increased productivity, as measured by issues resolved per hour, by 14 percent on average. With the greatest impact on novice and low-skilled workers, and minimal impact on experienced and highly skilled workers. The researchers attributed this effect to several factors:

• The AI assistant reduced the time spent on typing and searching for information, allowing the agents to handle more issues in less time.
• The AI assistant improved the quality and consistency of the responses, leading to higher customer satisfaction and fewer escalations.
• The AI assistant reduced the cognitive load and stress of the agents, enabling them to focus on more complex and challenging issues.

The researchers also found that the AI assistant improved customer sentiment, reduced requests for managerial intervention, and improved employee retention.

They measured these outcomes using natural language processing techniques to analyze the tone and sentiment of the customer messages, as well as administrative data on agent turnover and performance reviews.

How the AI assistant disseminates knowledge and helps learning

The researchers also provided suggestive evidence that the AI assistant disseminated the potentially tacit knowledge of more able workers and helped newer workers move down the experience curve. They argued that the AI assistant acted as a complement rather than a substitute for human workers, enhancing their capabilities and reducing their skill gaps.

The researchers tested this hypothesis by comparing the effects of the AI assistant across different groups of workers based on their tenure and skill level. They found that:

• The AI assistant had a larger effect on novice workers than on experienced workers, suggesting that it helped them learn faster and overcome their initial difficulties.
• The AI assistant had a larger effect on low-skilled workers than on high-skilled workers, suggesting that it helped them improve their performance and catch up with their peers.
• The AI assistant had a larger effect on workers who were exposed to more diverse and complex issues than on workers who faced more routine and simple issues, suggesting that it helped them expand their knowledge base and problem-solving skills.

The researchers also found that the effects of the AI assistant persisted over time, indicating that it did not induce complacency or dependency among the workers.

Implications for policy and practice

The study is one of the first to examine the effects of generative AI on the labor market, especially in the context of non-traditional work arrangements such as online platforms. The study has implications for policy and practice, as it suggests that generative AI can be a valuable tool for improving worker productivity and performance, as well as customer satisfaction and loyalty.

The study also highlights some of the challenges and opportunities for designing and implementing generative AI systems in real-world settings. For instance:

The study was authored by Erik Brynjolfsson, Danielle Li and Lindsey R. Raymond and was published as a working paper by the NBER in April 2023. The paper can be accessed at https://www.nber.org/papers