As artificial intelligence increasingly guides critical decisions in areas like lending and hiring, a fundamental question arises: how much should we trust these systems, and when should human judgment prevail? This research delves into the complex interplay between human oversight and AI, exploring how these collaborations impact fairness. The core inquiry centers on understanding the conditions under which we might either over-rely on potentially flawed AI recommendations or, conversely, dismiss valid AI advice due to our own biases. By examining these dynamics, this work seeks to provide actionable insights for designing oversight mechanisms that promote equitable outcomes in high-stakes AI-assisted scenarios.
Contextualize the findings concerning the use of AI in lending and hiring scenarios.
This study’s findings regarding AI’s impact on discrimination in lending and hiring must be understood within the broader policy landscape. The EU AI Act, for example, mandates responsible AI use through human supervision and emphasizes non-discrimination. Article 10 obligates AI system providers to implement robust data governance, examining datasets for biases and adopting mitigation measures. This policy relevance informs our study’s exploration of human oversight in AI decision-making, underscoring the need for continuous monitoring, post-deployment review, and the capacity to override AI decisions. The study’s examination of AI’s effects, particularly concerning bias and discrimination, are vital inputs to the AI Act’s bias identification and mitigation goals.
Furthermore, the research presented here reveals that AI-supported decision systems combined with human oversight can both perpetuate and mitigate biases—a nuance with significant context within broader ethical considerations of deploying high-risk AI. Current policies often rely on the assumption that human oversight will rectify AI shortcomings, which this study challenges by demonstrating that human biases may endure or even amplify when interacting with AI recommendations, even “fair” ones. From this perspective, current regulatory interventions and policy instruments should be critically reassessed to incorporate the potential biases that human oversight can introduce. Policy responses should also be proactive, considering feedback mechanisms, improved bias detection tools beyond AI testing, and effective human-AI collaborative frameworks to complement human input instead of simply replacing or imitating it.
Practical Implications and Future Considerations
The real-world implications of these findings extend beyond policy mandates into considerations of ethical AI design. Interventions such as organizational norm reviews, transparent override justifications, and continuous outcome monitoring are crucial. Decision-makers need access to insights into their performance and biases, along with the tools to explain their overrides, which promotes a proactive ecosystem of mutual checks. This holistic strategy aims to optimize AI-human collaboration so that decisions consistently meet both organizational objectives and ethical considerations, including robust protections against discriminatory practices. By framing oversight within realistic scenarios and fostering a systemic perspective, future regulatory sandboxes for AI can explore data governance, establish best practices, and promote public trust in systems used for lending and hiring.
What are the key research questions?
This research seeks to understand the intricate balance between trust and control in human oversight of AI decision-making. As AI systems become more prevalent, particularly in high-stakes areas like lending and hiring, it is crucial to determine the appropriate level of reliance on these systems. Key questions revolve around identifying scenarios of “over-reliance,” where users unquestioningly accept AI recommendations even when they are flawed, and “under-reliance,” where users dismiss valid AI advice in favor of their own judgment. The overall goal is to ascertain how such reliance behaviors impact the fairness of AI-assisted decision-making and to derive insights for designing effective oversight mechanisms.
A central research question is whether providing a theoretically unbiased AI translates into less biased decisions by users. This explores whether users trust and appropriately utilize fair AIs. Conversely, the study asks whether users maintain their ability and willingness to understand and question AI, even when it is theoretically unbiased. This probes the potential for “algorithm aversion,” where users reject algorithmic decisions in favor of their own, potentially biased perspectives. The research aims to determine whether biased overriding during ex-post oversight might negate the benefits of ex-ante efforts to ensure AI fairness, and conversely, whether ex-post oversight can mitigate the impact of failures in ex-ante oversight.
Ultimately, the study seeks to determine how user preferences impact their decisions to follow or reject AI advice, and whether AI recommendations can amplify the impact of discriminatory tendencies. In essence, it asks whether AI support can exacerbate or perpetuate discrimination, even when the AI itself is designed to be fair. By exploring these questions, the research aims to provide insights into designing human oversight systems that minimize discriminatory outcomes and maximize the benefits of human-AI complementarity in sensitive decision-making contexts.
What is the methodology employed in this research?
This research adopts a mixed-methods approach, employing a sequential explanatory design. This design involves an initial phase of quantitative data collection and analysis, followed by a subsequent phase of qualitative data collection and analysis, with the qualitative phase serving to explain and elaborate upon the quantitative results. The quantitative component consists of a behavioral experiment conducted online, simulating employer-employee and lender-borrower relationships with professionals in human resources and banking in Italy and Germany (N=1411). Participants made decisions about hypothetical applicants, with incentives tied to the performance of their chosen candidates, mimicking real-world professional assessments.
Following the quantitative experiment, a qualitative phase was undertaken, comprising semi-structured interviews and small-group online workshops with a subset of study participants and a co-design workshop with experts and policymakers. The interviews sought to explore participants’ experiences with AI in their professional contexts, their decision-making processes, and their perceptions of fairness and bias. The workshops served to further explore the ecological validity of the study, gathering feedback on how the selected candidate characteristics and experimental set-up related to real-life situations, and to generate new ideas on how to mitigate human and algorithmic biases. This triangulation of quantitative and qualitative data provides a comprehensive and nuanced understanding of the impact of human oversight in AI-assisted decision-making.
To enrich the analysis and derive actionable insights, participatory design methods were incorporated. A co-design workshop brought together experts from diverse disciplines to ideate on mitigating human and algorithmic biases in AI-supported decision-making. The experts discussed topics including defining algorithmic and human fairness, translating fairness into practical rules, regulatory requirements for oversight, and fostering awareness among users and developers. This diverse engagement allowed the formulation of grounded policy recommendations and future-oriented research priorities stemming from this large-scale study about the proper way to implement human oversite systems to avoid discriminatory outcomes. Finally, policymakers were invited to reflect on the findings and discuss policy implementations.
What are the primary findings of the quantitative experiment?
The quantitative experiment, a large-scale behavioral study involving HR and banking professionals in Italy and Germany (N=1411), yielded several noteworthy findings regarding the impact of human oversight on discrimination in AI-based decision support systems. A central result was that human overseers were equally likely to endorse discriminatory recommendations from a generic AI as they were to follow the suggestions of an AI specifically programmed for fairness. This starkly illustrates that human oversight, in its current implementation, does not inherently prevent discrimination when a potentially biased generic AI is utilized as part of the decision-making process. Furthermore, the study discovered that when a “fair” AI was employed, the choices made by participants exhibited less gender bias. However, this did not entirely eliminate discrimination, as choices remained influenced by the individual biases held by the participants themselves, showcasing the persistence of human biases despite the presence of seemingly unbiased algorithmic support.
Specifically, the generic AI, optimized for accuracy and exhibiting tendencies to favor men and German applicants, was observed to exert influence, leading to discriminatory choices against women and Italian applicants. Conversely, the “fair” AI, designed to minimize bias, appeared to mitigate discrimination against men but did not fully negate existing prejudicial attitudes. This suggests that while fair AI can nudge decisions towards less discriminatory outcomes, it cannot completely eradicate deeply ingrained human biases. Notably, the experiment also allayed some concerns, demonstrating that the mere presence of AI, even when programmed for fairness, did not necessarily augment the impact of discriminatory tendencies. The study found that individual preferences did not have more of an influence on choices when there was AI support.
Additional findings
Beyond the principal aspects of bias, the experiment also assessed the performance of deciders with and without AI support. Intriguingly, access to AI recommendations, whether from a fair or generic AI, did not demonstrably improve the quality of human decisions in terms of the intended objectives. However, the experiment did show that following the AI recommendations would have earned the decider more points compared to decisions being based entirely on unaided human intuition.
What are the main themes emerging from the post-experimental qualitative studies?
The post-experimental qualitative studies, comprising individual interviews and collaborative workshops with both study participants and AI ethics experts, delved into several key themes surrounding human oversight in AI-assisted decision-making. A prominent theme was the often-unacknowledged influence of organizational norms and perceived organizational interests on decision-making. Participants frequently expressed prioritizing company strategies and objectives, even when those priorities conflicted with individual fairness considerations or the recommendations of a “fair” AI. This suggests a need for interventions targeting organizational culture and providing clearer guidance on when and how to override AI recommendations in favor of ethical principles. The qualitative data also illuminated a tension between participants’ aspirations toward AI-driven efficiency and their practical hesitations about relinquishing “soft” qualities assessments to algorithms, especially concerning nuanced attributes identified during interviews.
Linked to the organizational influence was the recurring theme of *contextual factors*, which participants identified as playing a substantial role in their interaction with the AI and their trust with AI recommendations. This highlighted the limitations of the experimental scenario’s simplified, decontextualized view of candidates. Participants emphasized their need for a holistic understanding of a candidate’s circumstances and motivations, qualities they believed AI could not adequately capture. The desire for *explainability and justification* also emerged as a defining feature of effective human oversight. Participants needed to understand the reasoning behind the AI’s recommendations not only to reinforce their own judgment and expertise, but also to ensure transparency and accountability towards candidates. Some participants even expressed a reluctance to adopt certain AI systems because the way decisions were taken was not transparent. The combined study results suggest the importance of a human-centric approach to AI implementation, where technology serves as a tool to facilitate human interpretation and assessment rather than simply automating decision-making.
Finally, a recurring theme was the critical role of outcome feedback and continuous monitoring in refining both AI systems and human decision-making processes. Study participants expressed a need for feedback mechanisms that would enable them to assess the accuracy and fairness of AI-assisted decisions over time. A recurrent desire for both the professional in the field as well as for the fairness in AI experts that were consulted, was the accessability of a “test and audit system” that is accessible to all parties and where new learnings of the systems and the decision making proccess can be obtained. Such systems that “provide a clear” account of what is been seen and how will allow every one to engage in a fairer way by building that “communal engagement” which would further help by creating processes to enhance the long-term outcome of the system being used.. This focus on continuous learning highlights the need for AI oversight systems that can adapt to changing social contexts and facilitate collaborative refinement by incorporating human judgements. Experts proposed interventions at the training and operational level of AI development, to consider more nuanced and contextual data over more generic and potentially biasing or discriminating data sets.
How do the individual interview results inform the research questions?
The individual interviews provided valuable insights into the priorities and biases that professionals consider when making decisions, both related to fairness and pragmatic considerations. A key takeaway was the realization that participants often contextualized their decisions based on their personal experience and the strategic goals of their companies. Some admitted to accepting discriminatory recommendations from AI systems if they were aligned with organizational objectives, thereby highlighting a tension between ethical principles and practical demands. Furthermore, participants expressed a preference for AI decision support when tasks were less complex, such as data processing, and expressed hesitancy when judging nuanced human characteristics. The need for explainability and transparency was emphasized, as participants requested insights into the AI’s reasoning to feel reassured regarding the fairness.
The interviews shed light on the factors driving the willingness of professionals to use AI support, ultimately addressing a key research question. Participants’ openness to AI-assisted decisions hinged on several factors, including the perceived complexity of the task, the potential risks of relying on AI, and their ability to justify and explain the AI’s recommendations. The interviews also revealed nuanced views on AI’s helpfulness, with some participants considering it indespensible tools of labor and efficiency. Conversely, it was explained that many AI tools are too immature for some responsibilities, and this view was emphasized when explaining AI tools ability to assess human qualities and interrelate the importance of various qualities when making informed decisions.
Ultimately, the individual interview findings strongly suggested the need to assess and address biases that are not just contained within the AI system alone. The focus should also center on what external processes and biases exist in other members of the decision making process, including societal, human, or institutional. This informed what types of data should be collected in focus groups, and what topics required further detailed evaluation from external teams working in ethical and equitable AI development, such as the interdisciplinary workshop on AI justice. The insights gleamed from the individual interviews allowed researchers to shift their attention outside the AI black box, and understand that more human understanding and reflection would influence ethical and equitable decision support.
What insights were derived from the workshops with professionals?
Qualitative research involving workshops with professionals who participated in the human-AI interaction experiment revealed key insights into the complexities of algorithmic fairness and human oversight. Participants underscored the critical importance of contextualizing the use of AI within their specific professional scenarios. Furthermore, they emphasized the need for explainable AI, wanting transparency into the algorithms’ rationale and how decisions were made. This explainability, they argued, would foster greater trust in the AI system and enable better informed decision-making. However, participants noted a tension between algorithmic recommendations and their own professional judgment, often valuing their ability to assess nuanced, situation-specific attributes that AI may struggle to capture.
Another significant insight derived from the workshops was the notion of fairness beyond just non-discrimination. The experts highlighted that while achieving fairness in AI requires a multidisciplinary approach, involving both social and technical considerations, fairness is both a systemic and dynamic process, and is more contextual. Building collaborative fairness, according to the participants, requires awareness of when the system is failing and when humans are failing for the system. Furthermore, participants highlighted the importance of recognizing and addressing their own implicit biases, as well as the potential for organizational biases to influence decision-making processes. Participants expressed a need for clear guidelines on when to override AI recommendations and for mechanisms to review and monitor decisions to override, which helps mitigate unintended biases and promoting more equitable outcomes as they are guided to align their choices with organizational values and long-term goals. The workshops confirmed the value placed on individual decisions when the context is more detailed, while acknowledging that reliance could happen if results are validated or if organizations have limits.
What perspectives and conclusions were established by the experts’ workshop?
The experts’ workshop convened a multidisciplinary group to address biases in AI-supported human decision-making, focusing on scenarios such as hiring and lending. Crucially, the workshop identified defining fairness as a dynamic process requiring continuous practice and exercise, rather than a static concept. This perspective underscored the necessity of embedding fairness throughout the entire AI system development lifecycle, from initial data collection to ongoing implementation. The experts emphasized considering fairness in a dynamic and iterative manner in relation to real-world outcomes, including in the aftermath of a person in human oversight being “in the loop.” Discussions questioned the effectiveness of solely excluding protected characteristics from datasets to ensure fairness, with experts noting this, in and of itself, won’t guarantee it. This is because isolating the characteristics for analysis could also lead to misinterpretations or the complete loss of context.
The workshop further explored the operationalization of fairness in human-AI collaboration, highlighting mutual transparency as a central tenet. Experts proposed mechanisms for standardized control and feedback loops to promote ethical and regulatory oversight. Experts envisioned a future where AI could be incorporated as part of dynamic learning loops, or even contribute positively to a hybrid human and AI model in the work place or lending environment. The process could involve a team that performs and iterates through testing. For example, AI could be part of a team that examines both the technical and societal aspects of the project and can identify any issues. Experts also pushed for providing meaningful analysis and transparency toward the candidates who go before a human in oversight, as well as the process behind human oversight decisions. More specifically, a key theme consistently raised was the importance of fostering “AI ethics literacy” among both designers and users to ensure responsible AI system development.
The experts highlighted the significance of community involvement and shared responsibility in AI governance, advocating for collective forms of human oversight. Discussions emphasized prioritizing community needs over purely technical capacities, building new imaginaries to better support and more closely-link tech and community needs. From a policy perspective, the experts urged the EU to prioritize values and ethics in AI development and invest in building trust surrounding technology, ultimately helping to ensure that AI aligns with overarching strategic goals. Workshop participants also noted the role of feedback from end user-level decisions and AI-led decisions and suggestions in a larger ecosystem can also help lead researchers and developers to a system-level understanding of complex issues.
What are the principal discussion points and policy recommendations derived from the integration of the research findings?
The integration of findings from both quantitative experiments and qualitative interviews, combined with expert and policymaker workshops, highlights several crucial discussion points and informs specific policy recommendations regarding human oversight of AI-supported decision-making. A key overarching theme is the need to move beyond a simplistic view of human oversight as an automatic corrective mechanism for AI bias. The research underscores that human biases and organizational norms can significantly influence and even exacerbate discriminatory outcomes within AI-assisted systems. Discussion points center on the complexities of balancing algorithmic efficiency with ethical considerations, particularly regarding fairness and transparency in often-opaque decision-making processes. The study also reveals the importance of context and the limitations of a one-size-fits-all approach to AI oversight, emphasizing the necessity to incorporate diverse perspectives and address potential systemic issues alongside technical aspects.
Based on these discussion points, several policy recommendations emerge to enhance the effectiveness of human oversight. There is a need to not only mitigate AI biases, but also implement robust mechanisms for monitoring and reviewing human decisions that override AI recommendations. This would require investment in bias detection tools and training programs to increase awareness and knowledge of human decision biases. Transparency needs to be a key element, enabling stakeholders to gain insights on how final decisions get made. Furthermore, AI governance frameworks should consider including ongoing feedback loops allowing decision-makers to adjust AI algorithms based on real-world performance while adhering to ethical guidelines. These recommendations collectively aim to create a more holistic, context-aware and adaptive AI oversight system that promotes fairness.
Actionable Insights for Regulation
To make policies more effective, it is important to look at how human biases and organizational culture might affect the fairness of AI systems. Policymakers should consider regulations and guidelines that prompt organizations to check their culture and norms and to take measures that ensure AI systems and their human overseers do not reinforce existing inequalities. Further, policies should encourage AI developers to provide tools that enable transparency and justify explanations for the AI decision-making to the people affected as well as give context regarding what AI system’s data and decision-making does. It should be possible to create policies that encourage innovation while, at the same time, have safeguards to ensure AI systems support fairness and protect individual rights.
These findings challenge the simple assumption that human oversight inherently corrects AI shortcomings. Our research demonstrates that existing biases, amplified by organizational pressures and imperfect understanding of AI logic, can undermine even the fairest algorithms. Current policies must evolve to address these complex human-AI interactions, moving beyond technical fixes to include robust mechanisms for monitoring human override decisions, fostering algorithmic literacy, and promoting truly collaborative frameworks. This requires a shift towards a more contextualized, adaptive, and ethically grounded approach to AI governance, ensuring that these powerful technologies serve to promote fairness and equity rather than perpetuate existing inequalities in lending and hiring practices.
