When software screens job seekers
How employers, regulators and courts check automated hiring tools for bias
IAPQB Report No. 5 · Practice report · Published 17 November 2026
International AI Practice Qualification Board (IAPQB)
In one minute
- In 2018 Reuters reported that Amazon had dropped an experimental résumé-scoring tool. Trained on mostly male résumés, it marked down those containing the word "women's" [1].
- In 2023 iTutorGroup agreed to pay $365,000 to settle a US Equal Employment Opportunity Commission (EEOC) case over software that rejected older applicants [2]. In 2025 a US court let an age claim against the vendor Workday proceed, for now, as a group case [3].
- Since July 2023 New York City has enforced a law requiring yearly bias audits of automated hiring tools [4]. A 2024 study found audit reports from 18 of 391 employers; state auditors found at least 17 possible breaches where the city found one [5, 6].
- Regulators now ask for testing by group, independent auditors, notice to candidates and employer responsibility for bought tools [7, 8, 9]. EU rules for high-risk hiring tools apply from 2 December 2027 [10, 11].
1. A familiar situation
A recruiter opens the applicant list for a new post. A screening tool has already ranked several hundred people. The recruiter can see each score, without knowing how it was made. If the ranking pushes women or older people down, nothing on the screen shows it, and those at the bottom will not know why.
2. Why it happens
A screening tool follows rules set by people or learns from past data. If past hiring favoured one group, the tool can learn the signs of that group and repeat the pattern. In effect, Amazon's system "taught itself that male candidates were preferable" [1]. Amazon made the programs neutral to particular terms, but its sources said this was "no guarantee" against other discriminatory sorting [1]. A tool also applies one rule to every applicant, so one fault can reach many people: more than 200 in the iTutorGroup case [2]. Few complaints reach the regulator: New York City's consumer agency received two about such tools in two years, and state auditors found its complaint process ineffective [6].
IAPQB view. In the cases below, software sorted many applicants at once, or allegedly did, and the applicants could not see how. Amazon's own specialists found its problem, a federal agency sued iTutorGroup, and state auditors found the gaps in New York's audits [1, 2, 6]. In the Workday case, the court pointed to the number and timing of the rejections [12].
3. Four cases
3.1 Seattle, 2018: a résumé scorer that learned from past hiring
From 2014, an Amazon team built a tool that scored job applicants' résumés from one to five stars, people familiar with the effort told Reuters [1]. By 2015 the company saw that the system was not rating candidates for technical jobs in a gender-neutral way. It penalised résumés that included the word "women's", as in "women's chess club captain", and downgraded graduates of two all-women's colleges [1]. Amazon disbanded the team by the start of 2017; it declined to comment on the tool [1].
3.2 New York, 2023: software that rejected older applicants
The EEOC sued iTutorGroup for age discrimination. It said the company's tutor application software automatically rejected women aged 55 or older and men aged 60 or older, turning away more than 200 qualified US-based applicants because of age [2]. In a settlement announced on 11 September 2023, the company agreed to pay $365,000 to those applicants. It had already stopped hiring tutors in the United States; in case it resumes, the settlement requires training for hiring staff and bars asking applicants for birth dates. The EEOC will monitor compliance for at least five years [2].
3.3 California, 2025: a group case against the vendor
In Mobley v. Workday, a job seeker over 40 says that since 2017 he has applied for more than 100 jobs at companies using Workday's screening features, and was rejected each time [3]. He says Workday's AI recommendation system "can score, sort, rank, or screen" applicants [3]. Four other plaintiffs over 40 have joined him [3]. On 16 May 2025, federal judge Rita Lin gave preliminary approval for the age claim to proceed as a collective action, a group case that eligible people can join. The "critical issue", she wrote, is whether the system "has a disparate impact on applicants over forty": whether, in effect, it screens them out more often [3]. Workday argued that it makes no employment recommendations [3]. The Civil Rights Litigation Clearinghouse lists it as ongoing, with the docket last updated on 30 October 2026 [12].
3.4 New York City, 2023 to 2025: an audit law in practice
Since 5 July 2023, New York City has enforced Local Law 144 of 2021. Employers may use an automated employment decision tool only after a bias audit within the past year, with information about the audit made public and notice given to candidates [4]. The law "does not require any specific actions based on the results" [7].
In a 2024 study, 155 students from a Cornell University class searched the websites and LinkedIn and Indeed profiles of 391 employers. Of these, 18 had posted audit reports and 13 had posted notices to job seekers [5]. The authors warn that a missing audit does not prove a breach, because employers decide whether their tool falls under the law. They call this "null compliance" [5].
A New York State Comptroller audit, published on 2 December 2025, examined the city's enforcement from July 2023 to June 2025 [6]. The city's consumer agency had surveyed the websites and bias audits of 32 companies and found a single issue of non-compliance. Reviewing the same companies, state auditors found at least 17 instances of potential non-compliance, including five audits with incomplete calculations and three employers without an independent auditor [6]. The agency said it was "willing to adopt many" of the recommendations but disagreed with many findings [6].
4. How practitioners handle it now
Results are tested by group, by someone independent, every year. New York requires selection or scoring rates and impact ratios by sex, race and ethnicity, and their intersections, from an audit no more than a year old. An auditor who helped use, develop or distribute the tool, or has a direct or material indirect financial interest in the employer or vendor, is not independent [7].
Group data comes from candidates. The city's guidance says "imputed or inferred data cannot be used" for a bias audit [7]. In audits of AI recruitment tool providers, the UK Information Commissioner's Office (ICO) found tools that inferred gender and ethnicity from names, without asking candidates. It told the providers to collect accurate information directly from candidates and to check regularly for possible discrimination [8].
Candidates are told in advance. New York requires notice 10 business days before use, naming the job qualifications or characteristics the tool will assess and explaining how to request a reasonable accommodation [4, 7].
The employer stays responsible for a bought tool. In New York a vendor can commission the audit, but the employer remains "ultimately responsible" for making sure it was done [7]. In October 2026 three city agencies, including the consumer agency, added that an entity "cannot avoid responsibility" for breaking the law by relying on "an algorithm, AI, or other technology" [9].
In the EU, high-risk rules apply from December 2027. The EU AI Act classes AI systems that filter job applications or evaluate candidates as high-risk [11]. Under the AI Omnibus, in force since 27 July 2026, the high-risk rules for these systems apply from 2 December 2027 [10].
5. What to take from this
For recruiters and hiring managers. A screening tool can learn an old pattern from past hiring, and at Amazon the company's own specialists found it [1]. Staff who work with the rankings can see patterns that applicants cannot.
For HR leaders who buy these tools. Buyers can ask what the state auditors asked: who did the audit, on what data, and whether every required group and tier appears [6]. In New York the legal duty stays with the employer [7, 9].
For training providers and employers. State auditors found that officials at the agency enforcing New York's law lack technical expertise to evaluate these tools [6]. HR staff need enough skill to read a bias audit and spot its gaps [6, 7].
6. Limits of this report
All four cases involve US companies or authorities, and rules elsewhere differ. The Amazon account rests on unnamed sources [1]. The EEOC case was settled without a trial [2], and the Workday claims are allegations not yet proven [3, 12]. The study counted only what employers posted online; this report uses its conference version [5]. This report does not test any hiring tool.
References
[1] Reuters (10 October 2018). Amazon scraps secret AI recruiting tool that showed bias against women. As published by Arab News. https://www.arabnews.com/node/1385386
[2] US Equal Employment Opportunity Commission (11 September 2023). iTutorGroup to Pay $365,000 to Settle EEOC Discriminatory Hiring Suit. https://www.eeoc.gov/newsroom/itutorgroup-pay-365000-settle-eeoc-discriminatory-hiring-suit
[3] Mobley v. Workday, Inc., No. 3:23-cv-00770-RFL (N.D. Cal.), Order Granting Preliminary Collective Certification, Dkt. No. 128, 16 May 2025. Copy held by the Civil Rights Litigation Clearinghouse. https://clearinghouse-umich-production.s3.amazonaws.com/media/doc/166151.pdf
[4] New York City Department of Consumer and Worker Protection. Automated Employment Decision Tools (AEDT). https://www.nyc.gov/site/dca/about/automated-employment-decision-tools.page (accessed 16 November 2026)
[5] Wright, L., Muenster, R. M., Vecchione, B., Qu, T., Cai, P. (S.), Smith, A., COMM/INFO 2450 Student Investigators, Metcalf, J., & Matias, J. N. (2024). Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability. The 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT '24), 3-6 June 2024, Rio de Janeiro, Brazil. https://facctconference.org/static/papers24/facct24-113.pdf
[6] Office of the New York State Comptroller (2 December 2025). Enforcement of Local Law 144 – Automated Employment Decision Tools. Report 2024-N-6. https://www.osc.ny.gov/state-agencies/audits/2025/12/02/enforcement-local-law-144-automated-employment-decision-tools (full report: https://www.osc.ny.gov/files/state-agencies/audits/pdf/sga-2026-24n6.pdf)
[7] New York City Department of Consumer and Worker Protection. Automated Employment Decision Tools: Frequently Asked Questions. https://www.nyc.gov/site/dca/businesses/faq-automated-employment-decision-tools.page (accessed 16 November 2026)
[8] Information Commissioner's Office (6 November 2024). ICO intervention into AI recruitment tools leads to better data protection for job seekers. https://ico.org.uk/about-the-ico/media-centre/news-and-blogs/2024/11/ico-intervention-into-ai-recruitment-tools-leads-to-better-data-protection-for-job-seekers
[9] New York City Department of Consumer and Worker Protection (1 October 2026). DCWP, TLC, and CCHR Issue Enforcement Policy Statement on Artificial Intelligence. https://www.nyc.gov/site/dca/news/067-26/dcwp-tlc-cchr-issue-enforcement-policy-on-artificial-intelligence
[10] European Commission (27 July 2026, last updated 31 July 2026). AI Omnibus enters into force. Shaping Europe's digital future. https://digital-strategy.ec.europa.eu/en/news/ai-omnibus-enters-force
[11] European Commission, AI Act Service Desk. ANNEX III [of the EU AI Act: "High-risk AI systems referred to in Article 6(2)"]. https://ai-act-service-desk.ec.europa.eu/en/ai-act/annex-3 (accessed 16 November 2026)
[12] Civil Rights Litigation Clearinghouse. Mobley v. Workday, Inc. 3:23-cv-00770 (N.D. Cal.). https://clearinghouse.net/case/44074/ (accessed 16 November 2026; docket last updated 30 October 2026)
How to cite
IAPQB (2026). When software screens job seekers: how employers, regulators and courts check automated hiring tools for bias. IAPQB Report No. 5. International AI Practice Qualification Board. https://iapqb.org/research/005-hiring-tool-bias-audits.html
About this report
Prepared by IAPQB. Each fact comes from the sources listed, and each source was opened and checked on 16 November 2026; anything that could not be checked was left out. AI tools were used to search and to draft, and IAPQB checked every claim against its source and is responsible for the text. IAPQB runs an AI work qualification; this report does not promote it. Corrections: contact@iapqb.org or an issue at github.com/iapqb/iapqb.github.io. Licence: CC BY 4.0.
Archived with the series on Zenodo: doi.org/10.5281/zenodo.24248199 · Markdown sources: github.com/iapqb/research