Big boost! GPT-4 increased BCG employees' productivity compared to staff without the AI tool

Generative artificial intelligence and its impact on jobs and employment has been a constant debate ever since the new technology sprang up. The educated estimate is that AI is going to make some jobs obsolete, but it will also create new opportunities. However, one particular aspect where not much research has taken place is how AI and humans working together can impact work. This changed with a recent study that monitored the employees of the Boston Consulting Group (BCG) to find out the compounding effect of an AI-assisted working environment after handing some participants a GPT-4 powered AI tool.

The study was conducted by Harvard Business School and a preprint version of it has been uploaded to Social Science Research Network. The study, titled Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality, collected data from 758 BCG consultants. These participants were randomly assigned to one of three conditions — no AI access, GPT-4 AI access, or GPT-4 AI access with a prompt engineering overview.

BCG consultants with GPT-4 AI surpass those without it

Once the groups were assigned, all of the participants were given a set of 18 realistic consulting tasks. The study claims that the tasks were designed in a way that some of them would be easy for the AI to handle, while others were outside its current capability. The study found that consultants who used GPT-4 were significantly more productive than their counterparts.

The AI group completed 12.2 percent more tasks on average, which was completed 25.1 percent faster than the control group. Additionally, the AI group also produced more than 40 percent higher quality of tasks compared to the group without it.

“Consultants across the skills distribution benefited significantly from having AI augmentation, with those below the average performance threshold increasing by 43% and those above increasing by 17% compared to their own scores,” the study mentioned.

Interestingly, for a task selected to be outside the capabilities of the AI, however, consultants using AI were 19 percent less likely to produce correct solutions compared to those without AI, which highlights the downside of reliance on AI tools.

The study also found two different patterns people displayed while using AI for the given tasks. The study said, “One set of consultants acted as “Centaurs,” like the mythical halfhorse/half-human creature, dividing and delegating their solution-creation activities to the AI or to themselves. Another set of consultants acted more like “Cyborgs,” completely integrating their task flow with the AI and continually interacting with the technology”.