Managing AI Models’ Opacity and Risk Management Challenges
In today’s rapidly evolving technological landscape, AI models significantly impact decisions affecting millions of customers and billions of dollars daily. However, many financial institutions may lack the necessary understanding of these models to meet compliance requirements.
Key Insights
Opacity Challenge — AI models function fundamentally differently from traditional models. Unlike linear and traceable calculations, AI develops its own inferential logic, which model owners often cannot fully explain or predict.
Third-party Dependency Risk — Most traditional financial institutions utilize foundational models from external providers instead of building proprietary ones in-house. This reliance creates an additional layer of opacity, making validation and monitoring nearly impossible.
Regulatory and Trust Implications — Regulators globally demand transparency and control despite these limitations. The inability to explain AI decisions undermines customer trust, complicates compliance, and creates governance gaps.
The Challenge Explained
Developing customer-facing or internal models in the AI era is straightforward to comprehend but challenging to solve. Financial institutions create models to enhance decision-making, improve financial reporting, and ensure regulatory compliance. These models are employed across various banking operations, including credit scoring, loan approval, asset-liability management, and stress testing.
Traditional models, for which existing model risk management was designed, typically operate in a predictable, linear manner. Users can input data, trace calculations, validate assumptions, and confidently forecast outputs. This contrasts sharply with many AI applications, particularly those utilizing deep learning, where users struggle to predict outputs or explain inferences.
The Third-party Complication
The complexity increases as most financial institutions leverage foundational models from companies like OpenAI, Anthropic, and Google. These large language models (LLMs) serve as the backbone for applications ranging from customer service chatbots to risk assessments, introducing a new dimension of opacity.
Institutions face numerous model risk management implications, such as:
- How to validate a foundational model without access to its training data?
- How to ensure unbiased outputs when the inference process is not transparent?
- How to monitor model drift if the foundational model updates without notice?
Traditional vendor risk frameworks are inadequate for this level of dependency on opaque, constantly evolving systems.
When Traditional Risk Management Fails
Traditional model risk management relies on three components: initial validation, ongoing monitoring, and the ability to challenge model assumptions. Third-party foundational AI models disrupt all three components.
Initial Validation becomes problematic when institutions validate systems they can only observe externally. Unlike traditional statistical models based on explicit assumptions, AI models develop their own inferential logic through training, which may remain hidden.
Ongoing Monitoring presents similar challenges. Institutions relying on foundational models like OpenAI’s GPT must contend with updates that can alter performance without the institution’s input. Changes may compromise previous assumptions, making performance metrics difficult to measure.
Regulatory Landscape
Governments are implementing more detailed guidelines specifically targeting AI models. Financial institutions must demonstrate transparency and control over complex systems, including those sourced from third parties. For instance, in mid-2024, the Monetary Authority of Singapore issued guidance on AI model risk management, with similar initiatives emerging globally.
Real-world Consequences and Solutions
The stakes extend beyond regulatory compliance. If a model produces outputs understood only by an external team, operational risks can escalate. For example, customer service representatives may struggle to explain why a transaction was flagged by a fraud system, and loan officers must articulate the reasons behind credit model decisions. This opacity creates a trust deficit among customers and complicates compliance verification for regulators.
The industry is evolving with various responses. Some institutions are demanding greater transparency from AI providers, negotiating access to model documentation and performance metrics. Others are developing testing frameworks to validate third-party models through extensive input-output analysis.
Techniques like SHAP and LIME aim to clarify black-box decisions by approximating how models weigh different factors. Some institutions employ hybrid approaches, merging simpler, interpretable models with complex foundational models to balance performance and transparency.
The Path Forward
Financial institutions must integrate explainability and control mechanisms into their AI strategies from the outset. This may necessitate cross-functional teams of data scientists, risk managers, compliance officers, and vendor management specialists to negotiate terms with foundational AI providers.
Institutions also require comprehensive governance frameworks addressing the unique challenges of third-party foundational models. This could involve enhanced vendor due diligence, continuous monitoring, contractual provisions for model transparency and update notifications, and a willingness to forgo certain AI capabilities when risks cannot be adequately managed.
Despite these efforts, a fundamental tension persists: AI’s power partly stems from its ability to identify patterns at scale, often in ways we do not fully comprehend. When third-party providers are involved, predictability and control become increasingly elusive. Institutions must leverage the benefits of foundational models while acknowledging the uncertainties and limitations inherent in their use.
Successfully navigating these challenges can provide a strategic advantage. Institutions that harness third-party AI’s capabilities while maintaining oversight will excel, whereas those that fail to comprehend the associated risks may face severe repercussions in an industry where trust and compliance are paramount.
Understanding the complexities of AI model management is not just a regulatory requirement; it is essential for sustaining customer trust and operational integrity in the financial sector.
