Accountable AI in Government: The Role of Procurement
The integration of artificial intelligence (AI) into public governance has stirred significant debate, particularly around the mechanisms that determine how these technologies are acquired and implemented. A notable case study highlighted the use of predictive policing software by the New Orleans Police Department, which was provided by Palantir without any financial transaction. This scenario exposed critical flaws in the city’s procurement processes, raising questions about transparency and oversight.
The Procurement Loophole
Public procurement processes are designed to ensure that government contracts, whether for school buses or AI systems, undergo rigorous scrutiny. However, the New Orleans case illustrated a major loophole: since the software was a philanthropic gift, it bypassed standard checks that typically require city council approval and public debate. This lack of oversight resulted in key officials being unaware of the partnership, underscoring the need for reform in procurement laws to address such gaps.
Research Findings on AI Procurement Practices
In response to these issues, a research team comprising scholars from reputable institutions undertook an investigation into the purchasing processes that shape decisions surrounding public sector AI. Through interviews with city employees across seven anonymous U.S. cities, the research revealed that procurement practices vary significantly, influencing the governance of AI implementations.
The Importance of Procurement in AI Governance
Procurement serves as a critical lever for governments to enforce public values such as safety, non-discrimination, privacy, and accountability in AI technologies. However, current reform efforts are inadequate if they ignore the realities of how purchasing decisions are made. Successful AI procurement reforms must reconcile these goals with established purchasing norms.
Process of AI Procurement
Typically, when a government identifies a use case for AI, it initiates a solicitation process that includes inviting vendors to submit proposals through a Request for Proposal (RFP). City employees then evaluate these proposals through structured reviews, ultimately selecting a vendor and negotiating a contract. However, many AI systems circumvent this formal process, opting for alternative purchasing pathways.
Alternative Purchasing Pathways
Procurement laws often allow for small-dollar purchases to skip competitive bidding, enabling employees to acquire low-cost AI tools using government-issued purchasing cards. Additionally, AI technologies may be acquired through donations from companies, partnerships with universities, or simply accessed as free public resources. This tendency raises concerns about the effectiveness of current procurement frameworks in ensuring responsible AI governance.
Challenges in Centralized vs. Decentralized Procurement
The research identified a stark divide in how cities manage their IT portfolios. Some cities have centralized oversight where all software acquisitions must be vetted by IT staff, while others operate with a decentralized model, granting individual departments autonomy over their technology choices. This disparity suggests that a one-size-fits-all reform approach may not be viable.
Key Questions for Future Oversight
The findings prompt essential questions for local governments aiming to implement effective oversight for AI acquisitions:
- How can oversight processes be established for AI proposals that bypass conventional procurement methods?
- Who within the government is responsible for managing the risks associated with procured AI technology?
- How can existing procurement workflows be restructured to ensure meaningful evaluations of proposed AI solutions?
Conclusion: The Future of AI Procurement
As local governments increasingly recognize the potential of public procurement as a mechanism for implementing responsible AI, there is a pressing need for collaborative efforts among policy experts, researchers, and advocates. Effective AI procurement practices are crucial for ensuring that the technology serves the public good. Ultimately, understanding the procurement processes that lead to the adoption of AI systems is vital for assessing their impact on society.
