Sorry, AI-Driven Machine Learning Is Not an Easy Button for Record Classification

Let’s face it; getting employees to classify records is hard. Most employees do not want to spend time on record classification. As a result, unclassified emails, files, and other unstructured content continue to pile up. The digital clutter that builds from this reluctance drives up risk, lowers compliance, and drains productivity.

The temptation of the AI “easy button”

Many vendors now promise an “easy-button solution”: AI-driven machine learning. Machine learning (ML) offers the vision of automatically classifying emails and files as either records that must be retained or low-value, non-record information that can be deleted. ML, a form of AI that has been around for years, powers tools such as Microsoft 365 Purview’s trainable classifiers. Other platforms, including BigID and OneTrust, also promote ML-driven classification. With tools like these, it is tempting to believe ML may finally solve the problem of records sprawl.

In reality, while ML can successfully identify and classify certain types of information, it is not a practical strategy for classifying most records. The universe of records within most organizations is far larger and more varied than these tools can handle.

Where machine learning shines

ML excels at pattern recognition, which makes it effective for detecting specific kinds of personal or sensitive data. The AI component allows it to recognize a range of patterns. For example, ML can identify social security numbers whether they appear as a string of digits (123456789), include dashes (123-45-6789), or have spaces (123 45 6789). It can even recognize patterns based on nearby context, such as the words “social security number” followed by digits.

Many ML vendors provide a toolkit of preprogrammed “classifiers” that can identify common data types such as personal information, contracts, or credit cards. These typically work well for data that contains predictable formats or “regular expressions.”

Where machine learning falls short for records management

In practice, however, many records do not include clear keywords or consistent patterns. They often lack the structured cues on which ML models rely, which makes automated classification unreliable.

To address this, ML vendors promote “custom classifiers.” Microsoft Purview, for instance, allows users to train classifiers to recognize specific record types. The idea is that if the standard models cannot find a certain kind of record, a custom model can fill the gap.

This is where the challenge begins. In most cases, organizations can reach about 40% accuracy with standard models and another 10% with custom ones. After that, progress slows dramatically. Each additional improvement captures only small incremental gains. The first results can feel promising, but each step after that becomes exponentially harder.

Some tools boast the ability to create thousands of custom classifiers. However, building, testing, and maintaining even a few hundred of them is an enormous effort. For most organizations, this is neither scalable nor sustainable. A regulator will not be impressed by a claim that only 65% of records are classified. ML can be useful for certain tasks, but it is not a feasible core strategy for enterprise-wide records management.

One insurance carrier invested more than a year developing trainable classifiers, only to find that ML could automatically classify about half of their records. Another global manufacturer tried to teach a vendor to automatically find records but gave up after a year and a half.

A practical path forward

So, should we give up on solving over-retention? Absolutely not.

There is another approach that does work: data placement and automation. Instead of asking AI to determine what something is, this method teaches the content management system to apply the correct rules when content is stored in the right location. This is not “true” auto-classification, since it requires users to spend a few seconds selecting the proper folder, but it comes close. Once the content is placed, the rest of the governance process becomes fully automated.

This does not mean ML has no value in information governance. ML remains highly effective for identifying sensitive data, reducing redundant or obsolete content, and highlighting security risks, but it is not the ultimate solution for records classification. The key is matching the right tools to the right problems. For large-scale records classification, ML alone cannot get us there.

Eventually, generative AI may succeed where old-fashioned ML AI has struggled. The technology holds great promise, and it will likely become a game changer in the near future. That will be the focus of a future column.

Until then, the smartest path combines data placement and automation with selective, targeted use of ML. This approach reduces chaos today while laying the foundation for organizations to benefit from smarter automation tomorrow.