A study conducted by Stanford University and the Massachusetts Institute of Technology (MIT), which has focused on test the impact of generative AI tools on the productivity of a company for a yearhas shown that the customer service employees of said company with access to these tools improved their productivity, on average 14% compared to those who did not use them. Of the company that has participated in the study, it is only known that it is dedicated to software for SMEs in the United States, and that it is in the Fortune 500 index.
According to Bloomberg Technology, until now studies of the influence of large language models have only been conducted in fields such as law and medicine. These showed, for example, that GPT-4 has obtained high results in the entrance exam to the law in the United States, or the impact of technology on the performance of workers in isolated writing tasks in laboratory settings, to small scale.
The results of these previous experiments showed the potential of language models at work. But according to Erik Brynjolfsson, Director of the Digital Economy Lab at Stanford Human-Centered AI InstituteUntil the tools are tested in the real world, their impact is mostly speculative.
Thus, this researcher, who is one of the authors of the study, has highlighted that «getting people to use them for a year at this company you get a much better idea of how that translates to productivity in the real world. As far as I know, it’s the first time it’s been done in a real world environment.«.
Brynjolfsson, along with MIT researchers Danielle Li and Lindsay Raymond, has tracked the performance of more than 5,000 customer support agents, mostly based in the Philippines, on key metrics. Among them, the speed and success with which workers were able to solve customer problems. Groups of agents were created, and some were given access to Artificial Intelligence tools, trained on a large set of customer service conversations. Others were not given access.
One of the study’s findings was that new workers benefited the most from this technology. With the assistance of Artificial Intelligence, in addition, lower-skilled workers were able to do their work 35% faster. New worker performance also improved much faster with AI assistance than without.
According to the study, agents with two months of experience who were assisted by AI performed as well or better in many cases than agents with about six months of experience working without AI.
More favorable impact on new and less-skilled workers
Furthermore, the research suggests that the boost in worker productivity and performance may be due, in part, to the way AI tools can absorb the tacit knowledge that makes for top-performing workers in the company. stand out more.
Among this knowledge is knowing what type of language is best to use to calm an angry customer, or what technical documentation is the most useful to share in a given situation. With AI, this knowledge is passed on to less-skilled or less-experienced workers through suggested responses generated by Artificial Intelligence.
These findings go against what many have claimed that automation tends to do more harm to less-skilled workers, as has happened in recent decades with technological advances in manufacturing and other sectors. The increase in output was less noticeable than in previous experiments, probably because real-world processes are much more complex than direct tasks. Despite this, the improvement in productivity was significant.
Less-skilled workers received, however, less benefit from the entry of AI into their work. These, with higher performance, were already giving answers at the same level as those recommended by the AI, so there was less room for improvement.
The AI suggestions may have even been a distraction for them, the researchers suggest. And if AI succeeds in narrowing the gap between low-skilled and high-skilled workers, companies may need to rethink the logic they use to decide how much compensation their employees receive.
High-performing customer service agents have spreadsheets with collections of frequently used phrases that work well, according to Raymond. What an Artificial Intelligence tool does is take this knowledge and distribute it to the rest. «So these highly skilled workers are doing extra work for the company by offering these examples to the AI, but they are not getting compensated for it.«.
A change in the incentive and remuneration systems, necessary
In fact, the situation may be even worse, because your incentives are based on your performance relative to your peers, which raises many questions about new compensation policies related to compensation for the value of your data.
In any case, forward-thinking companies should recognize the expertise of their best employees, as their tactical knowledge and skills will likely form the foundation for the AI tools that will power the rest of the organization, according to Brynjolfsson.
For him, successful companies will have incentive and reward systems that recognize their best workers, whether or not their performance with a client proves to be better than less-skilled workers. This reward should be based on the knowledge they create for the entire company., and on which it depends. With this system, according to Brynjolfsson, one high-level worker can change the entire company.
Of course, the researcher acknowledges that it is still early to know the final impact of generative AI, and that there is still a lot to learn about it. Some of the observations from this experiment have not been captured by the data, but they point to many other ways of changing businesses and workplaces that these tools may facilitate in the near future.
Among them, that the area managers of the company did not spend 20 or 30 hours each week training employees with AI, because Artificial Intelligence replaced them to a certain extent. That could also change the boss-employee relationship, as supervisors would spend less time creating direct reports, and more time focused on building bigger teams.
Of course, the speed at which generative AI seems to be able to transform companies, something that seems like it can happen almost overnight, is dizzying, especially with respect to the speed with which previous advances have done it. . In any case, and in light of these preliminary results, Bryunjolfsson has some advice, both for workers and managers: adopt this technology.
For Brynjolfsson, you have to start with «experiment with it and learn what it can do. Discover where it is most effective and where it is least«. Furthermore, he notes, “companies should have crash programs to educate their staff about them and really accelerate«.
