Three Reasons Why Artificial Intelligence Wins With Specialization

When I began learning about artificial intelligence (AI) and machine learning (ML), it was from the perspective of trying to help very smart people do challenging tasks. This is opposed to learning about an AI algorithm and trying to see how far I could push it for the sake of science (which has its own merits) and then searching for an application in the real world. Some early examples of AI systems were for fire prevention on naval ships, predicting component failure causes in jet engines and classifying reasons for claim denial by health insurance companies.

Some lessons I learned were that smart people solve these problems every day, so an AI system needs to augment their intelligence and not provide a poor replacement for it. When we hear about AI systems like chatbots giving illogical replies, large language models failing with edge cases and image-generating systems creating nonsensical outputs, I believe we are witnessing a lack of AI specialization or use-case customization. Here are three reasons why specialization is needed in real-world AI systems.

1. Experts rely upon a few high-quality, specialized sources to make inferences.

In my experience fine-tuning AI language models for finance tasks, I’ve found that many AI models are increasingly trained using massive datasets. These models represent an achievement of both engineering and mathematics, and although they show increased performance with representative datasets, they rarely achieve the same kind of performance with most industrial applications.

The sheer volume of data means it can never be thoroughly cleaned or sampled for a specific use case. For example, my main focus is on using AI to understand the material relationships between companies (what drives their performance now and in the future). We need to develop a specialized fine-tuning approach using both thousands of labeled data as well as data labeled using an “adversarial” model.

Read more