The Growing Divide in AI: Large Models vs. Targeted Solutions
The Role of Experts in Large Language Models
In our last blog post we introduced the concept of experts, which play an important role within Large Language Models. LLM experts are not accomplished in a specific function, such as filling in tax returns, they are expert at one of the internal processes of the LLM. Selection of the correct expert for the job by the LLM results in a reduction in the amount of processing, and therefore time, to complete a task.
However, in some cases a neural network that is informed about a single function is precisely what the user requires. We present a short history of AI techniques and how this has led to interest in AI solutions that embed prior art within their structure.
Why More Data Means More Processing
Greater uncertainty creates a heavy computational burden. Problems that have precise mathematical models don’t need training data, because everything that is needed to formulate the solution is contained within the equations that describe the problem. In these cases, the correct answer can be reached quickly with relatively little computational effort.
However, as uncertainty about the solution to the problem increases, then so does the number of training examples needed to fill in the mathematical gaps. As a rule-of-thumb, every time the number of training examples doubles, the processing increases by a factor of ten.
The graph gives the approximate relationship between training data used and associated processing requirements for AI numerical methods introduced during the last fifty years.
The origin is 1975, when computational limitations precluded the use of training data to generate answers.
Recent EU AI legislation has set a boundary of 1019 MFLOPS during the training phase as the computational boundary above which all AI applications must be closely monitored. This boundary has been exceeded by many LLMs, including ChatGPT4.
The Challenge of AI Scalability for SMEs
By the end of 2024, it was widely accepted that the computational barrier required to develop an LLM, and many regenerative AI algorithms, would far exceed the resources available to an SME. This fact, along with a liberalisation of copyright laws, would have profound implications for all small businesses around the world.
The inexorable march to embrace applications with greater model uncertainty has produced a near-monopoly of providers. Computer scientists, wishing to push back against this outcome, have recently focused their efforts on methods of reducing the computational resources needed to build AI models.
