Refuting Bloomberg's analysis: ChatGPT isn't racist. But it is bad at recruiting.

🌈 Abstract

The article discusses the findings of a Bloomberg analysis that claimed to have found racial bias in OpenAI's ChatGPT language model when evaluating resumes. The author re-analyzed the data and found no statistically significant evidence of racial bias in ChatGPT's resume evaluations. However, the author's own testing revealed that ChatGPT tends to overestimate the performance of candidates with credentials from elite schools and companies, while underestimating the performance of candidates without such credentials. The article argues that this bias, while not based on race, is still problematic and highlights the limitations of using AI systems for hiring decisions.

🙋 Q&A

[01] Findings from Bloomberg's Analysis

1. What were the key findings of Bloomberg's analysis on racial bias in ChatGPT's resume evaluations?

• Bloomberg found that ChatGPT showed racial bias across all groups, except for retail managers ranked by GPT-4.
• Resumes with names distinct to Black Americans were the least likely to be ranked as the top candidates for financial analyst and software engineer roles.
• Names associated with Black women were top-ranked for a software engineering role only 11% of the time by GPT, 36% less frequently than the best-performing group.
• ChatGPT's gender and racial preferences differed depending on the job being evaluated.

2. How did the author's re-analysis of Bloomberg's data differ from Bloomberg's findings?

• The author re-ran the numbers using the same method as Bloomberg and found that there was no statistically significant evidence of racial bias in ChatGPT's resume evaluations for software engineers.
• The author found that GPT-3.5 did show racial bias for HR specialists and financial analysts, but GPT-4 did not show racial bias for any of the race/gender combinations.

[02] The Author's Own Testing of ChatGPT

1. What was the purpose of the author's own testing of ChatGPT's resume evaluation capabilities?

• The author wanted to run a sanity check on ChatGPT's resume evaluation abilities, using data from their own platform, interviewing.io, which hosts a large number of technical interviews.

2. What were the key findings from the author's testing of ChatGPT's resume evaluation?

• ChatGPT's resume evaluations were only slightly better than random guessing, with an AUC (area under the ROC curve) of around 0.55.
• ChatGPT consistently overestimated the performance of candidates with credentials from elite schools and companies, while underestimating the performance of candidates without such credentials.

3. Why did the author believe ChatGPT performed poorly on this task?

• The author suggests that there may not be much signal in resumes to begin with, and that ChatGPT exhibits a similar bias to human recruiters in overvaluing credentials from elite schools and companies.

4. What are the implications of these findings for the use of AI in hiring decisions?

• The author argues that off-the-shelf AI solutions like ChatGPT are not a magic pill for improving hiring decisions, as they exhibit similar flaws and biases as human recruiters.
• The author cautions that the widespread adoption of AI in hiring, especially among large enterprises and recruiting firms, is concerning given the limitations of these systems.