Does LLM AI Have Enough "Algorithmic Consistency" to Transform Lending? Part One
The AI Imperative in Lending
AI is so hot that every lender must address it and have a strategy. Every vendor that serves lenders is adding the word "AI" to every offering. But lending requires a more rigid application of AI than some of the more popular uses.
In this three-part series, we explore how lenders can make sure AI has enough "Algorithmic Consistency" to transform their lending operation.
- Part One: Challenges to Algorithmic Consistency
- Part Two: How Much Algorithmic Consistency is Necessary?
- Part Three: Testing for Algorithmic Consistency
What Is Algorithmic Consistency?
Algorithmic Consistency is simply the idea that a process consistently produces the same results from the same input.
Computers have been built on algorithms, but algorithms long pre-date computers. An "algorithm" is a sequence of steps that are applied to some input. It has the characteristic that it will consistently produce the same result. If the series of steps leads from A to B, then it will lead from A to B every single time the algorithm is executed.
Pre-AI, computer programs are algorithmic. They produce absolutely consistent output. For example, if you ask a calculator to produce the square root of 737,881, it will show 859 every single time. Or a lender that obtains a custom score from a MISMO credit report using LoanCraft's SCORE engine will consistently get the exact same custom credit score for any report that has the same credit characteristics.
Why Algorithmic Consistency Matters
Algorithmic Consistency is a gold standard of consistency. One of the great contributions of the computing age has been the algorithmic consistency of results. Credit scores are now relied upon universally in lending and have created a level of consistency in decision-making unmatched by human counterparts. This has led to scalability of lending, better risk management, and more fair lending.
Human reasoning falls short of it, and similarly, AI models fall short as well.
AI Large Language Models do not use algorithms in the same sense, and do not provide 100% algorithmic consistency. We know this from stories about AI "hallucinating" and getting simple math problems wrong. But as they develop, their consistency is improving, and they are getting closer to algorithmic consistency. The question for lenders becomes: are they consistent enough?
Why LLMs Fall Short of Algorithmic Consistency
Let's sketch a few reasons why LLMs do not provide algorithmic consistency.
- Large Language Models use pattern recognition, not algorithms. Answers are not generated by application of exact algorithms to inputs. This is part of the power of LLMs, because they are a way to generate results without specifying exact sequences of reasoning.
- Models "learn" and their learnings can change their answers. As models are updated over time, the same input may produce different output than it did previously.
- Answers can depend on availability of compute resources, and compute resources vary. You can test this by asking an AI model a question, and then asking it to think again. Answers will vary, especially for more complex questions. This is a decision made by the AI model for various reasons. People want answers quickly and shallower levels of analysis can provide quicker answers with less compute resources. There are also some more technical but important reasons why answers must vary over large sample sizes.
- Solution processes are not always transparent. AI does not necessarily "show its work" and the nature of neural networks and pattern recognition sometimes means that it's not plausible for the LLM to "explain" how it got an answer.
What This Means for Lenders
LLMs by their nature do not yield the same algorithmic consistency as pre-AI computer programs. For lenders considering AI, the question becomes: "What level of consistency is required?" We will explore this question in Part Two.
LoanCraft has been applying cutting edge technology to lending processes for over twenty years. Learn more at loancraft.net.