Do Your Master Records Have to Be Clean Before You Automate?

"We would have to sort out our data first before we take something like this on." That sentence turns up in nearly every first conversation, and as a general statement it is wrong. It postpones a project to a date that never arrives, because master records in a running business are never finished.

Something narrower is true. Three kinds of field genuinely hold a project up. Everything else can be surfaced during the automation and improved along the way, often better than in a clean-up beforehand, because by then you know which fields anyone actually uses.

The three fields that really matter

The first is unique keys. Customer number, article number, order number. If the same customer number exists twice, or a customer carries different numbers in two systems with nothing recording that they are the same, no automation can match anything. That is not a matter of effort. It is a hard limit.

The second is fields that carry several pieces of information at once. A field called Name rarely holds only a name. It holds salutation, title, first name, surname, company and occasionally a suffix such as a department or a ward, in varying order and with no separator that would let you take it apart again.

The third is status fields that carry meaning. Whether a customer is on hold, whether an article is discontinued, whether an address still applies. If that lives as a remark in a free-text note rather than in a field, an automation cannot evaluate it, and it will then make decisions without that information.

What a combined name field does

Suppose an import years ago split a single name field at the first space and spread the parts across first name and surname. For natural persons that mostly works. For a company such as Berger Medical Supplies Ltd, the first word lands in the first name and the rest in the surname. For a person with an academic title, the title lands in the first name.

As long as nobody uses those fields for a salutation, nothing shows. It becomes visible with the first mail merge, and it becomes visible across every affected record at once. That is the awkward property of data faults in automated processes. They do not appear one at a time. They appear in bulk and they appear in front of customers.

So these fields belong split apart before the first process uses them to send anything outward. Split means salutation, title, first name, surname and company name are separate fields, and a record is allowed to be a company with no person attached.

Why an automation should flag rather than repair

The obvious idea is to catch the fault with a clever rule. A rule that recognises a legal-form suffix as a company marker and adjusts the salutation accordingly. Rules like that work on a large share of records and are wrong on a small remainder, and that remainder is what goes out to customers.

The better construction is unglamorous. The automation applies the rule only where it holds with certainty. Anything it cannot classify with confidence gets a neutral salutation and additionally lands on a review list. That list is short and gets worked through once, instead of being guessed again on every send.

Just as important, the automation should not silently overwrite the source record. Corrections belong where the data is maintained. Otherwise you correct the same value on every run and eventually nobody knows which version is authoritative.

What poor master data costs inside a project

This is why the topic belongs in a conversation about price. Every field whose content is not reliable becomes a rule. Every rule has exceptions. Every exception is a business decision somebody in your building has to make and write down.

That is where fixed-price quotes acquire their buffer. A provider who has not seen your data either costs it cautiously, and you pay for the buffer, or costs it tightly and invoices the difference later. Looking at an export of two hundred records before the quote prevents both outcomes.

What you can check yourself in two hours

Export two hundred records from the system the process will use, and look at four things.

Do company names appear in person fields, or the reverse? Are there records that obviously mean the same customer under different spellings? Are fields empty that the process depends on, such as an email address for a process that sends email? And are important states sitting in free text instead of in a field?

That short summary is worth more in a quoting conversation than any requirements list, because it forces the provider to cost the real work instead of quoting an average from somewhere else.

The recommendation

Do not clean up all your master data. Clean up the fields that appear in the one process you automate first, and limit it to the three kinds above: unique keys, combined fields, states.

Everything else is better handled by an automation that flags uncertain cases instead of guessing them. That gives you two things at once: a process that runs, and a list of the records that genuinely need attention. That list is usually far shorter than the fear expressed in the first conversation.

Sophera Consulting reviews an export of your data before quoting, records which fields must be cleaned and which cases the process can simply flag, and then names a fixed price with no subscription. The entry point is free in the Automation Check.