Match rules and duplicate handling in Boomi DataHub
How to match customer records safely: exact vs fuzzy rules, quarantine for uncertain cases, stewardship to resolve them, and source ranking, built in Boomi DataHub.
A master data hub is only as good as its matching. Match too little and every duplicate survives into the golden records. Match too much and two different customers become one, with one’s invoices showing up in the other’s account. This article explains how to design match rules that balance the two, what to do with the uncertain cases, and how to set it all up in Boomi DataHub.
Why naive deduplication fails
The usual first attempt is “same name means same customer”. Look at what it does with real data from an ERP, a CRM and an e-shop:
| Source | Name | VAT number | Postcode | |
|---|---|---|---|---|
| ERP | Papadopoulos S.A. | 094123456 | info@papadopoulos.gr | 11526 |
| CRM | PAPADOPOULOS SA | 094123456 | g.papadopoulos@papadopoulos.gr | 11526 |
| E-shop | Papadopoulos Giorgos | g.papadopoulos@papadopoulos.gr | 11526 | |
| E-shop | Papadopoulou Maria | maria.p@gmail.com | 54622 |
Exact name matching links nothing: the spellings differ. Loose name matching (“starts with Papadopoul”) links all four, including Maria in Thessaloniki, who has nothing to do with this company. Neither is acceptable.
The pattern: layered rules plus a human in the loop
Good matching uses several rules in order of strength, and sends doubtful cases to a person instead of guessing.
- Strong identifiers first, exact match. VAT number, company registration number, or an email that is unique in your data. If they are equal, it is the same entity.
- Combined fuzzy rules as a fallback. “Name is similar and postcode is equal” is much safer than “name is similar” alone. Combining a fuzzy field with an exact one is the single most effective way to avoid false matches.
- Quarantine for ambiguity. If a record matches more than one golden record, or matches only weakly, stop and ask a data steward.
- Survivorship after the match. Once records are linked, decide per field which source’s value wins.
Exact vs fuzzy
| Kind | How it compares | Good for | Risk |
|---|---|---|---|
| Exact | Values are equal (often after normalising case) | VAT number, email, codes | Misses typos, variants |
| Fuzzy | Similarity score above a threshold | Names, street addresses | False matches |
Fuzzy matching is not magic. It scores how similar two strings are. “Papadopoulos S.A.” and “PAPADOPOULOS SA” score high; so, unfortunately, do “Papadopoulos Giorgos” and “Papadopoulou Maria”. That is why a fuzzy rule should almost always be combined with an exact condition.
Normalise before you match
Many “fuzzy” problems disappear if you clean the data first: trim spaces, unify case, remove legal suffixes (S.A., SA,
IKE, EPE, Ltd), format phone numbers as +30..., transliterate Greek and Latin consistently. Do this in the integration
process before sending, or in the model’s data quality steps.
Building it in Boomi DataHub
Match rules, data quality steps and sources are all defined on the model. Changes take effect when you publish a new model version and deploy it. Exact labels and available options can change between releases, so check the current documentation for your account.
Step 1: Add an exact rule on the strongest identifier
Open the Customer model (see
Building a customer golden record from ERP, CRM and e-shop with Boomi DataHub)
and go to its match rules. Add a first rule that matches when vat_number is equal. Records without a VAT number simply
don’t match on this rule and move on to the next one.
Step 2: Add a combined fuzzy rule
Add a second rule for records without a strong identifier, for example the e-shop’s private customers:
emailexact, ornamefuzzy with a fairly high similarity threshold andpostcodeexact.
DataHub lets you choose the comparison type per field in a rule, and for fuzzy comparisons a similarity algorithm and threshold. Start strict (higher threshold) and loosen it only after reviewing results; a missed match costs a steward a minute, a false merge can cost a customer.
Rules are evaluated in order, so put the most reliable rule first.
Step 3: Add data quality steps
Add data quality steps to the model for checks that should block bad records, for example a required name, or a
format check on vat_number (nine digits for Greek VAT numbers). Records that fail are quarantined instead of polluting
golden records. If you already normalise in the integration process, keep the model’s steps for validation only.
Step 4: Test with real samples
Before deploying, collect twenty or thirty tricky real cases: known duplicates, known non-duplicates that look alike, records with missing fields. Send them to a test repository and check the outcome of each. Keep that sample set; you will reuse it every time you change a rule.
Step 5: Work the quarantine
When DataHub cannot process a record safely, it places it in quarantine with a reason. Typical reasons include a record matching more than one golden record, a possible duplicate, a required field missing, or a failed data quality step. Data stewards review quarantined entries in DataHub’s stewardship screens and decide: link the record to an existing golden record, create a new golden record, or reject it and fix the data at the source.
Two habits make this sustainable:
- Fix at the source when possible. If the e-shop keeps sending a malformed postcode, change the e-shop or the process, not just the quarantined entry.
- Feed lessons back into the rules. If stewards keep resolving the same kind of case the same way, it is a candidate for a new rule or a normalisation step.
Step 6: Decide which source wins
Matching decides which records belong together; survivorship decides which value the golden record keeps. In DataHub this is governed by how sources contribute updates and, in current versions, by source ranking, which lets you say that some sources are more trusted than others for given fields. For our example:
| Field | Preferred source | Why |
|---|---|---|
name, vat_number |
ERP | Comes from invoices and tax data |
phone, contact data |
CRM | Sales update it after every call |
email (private) |
E-shop | Confirmed by the customer at sign-up |
Check how source ranking is configured in your DataHub version and whether it applies per field or per source; the behaviour is worth testing with your sample set before going live.
Before you go live
- Measure. Track how many records were matched, created and quarantined per run. A sudden jump in new golden records often means a rule stopped working (for example, a source began sending VAT numbers with a country prefix).
- Be careful with rule changes. New rules apply to incoming records; check how existing golden records are affected and whether existing duplicates need a one-off clean-up.
- Keep GDPR in mind. Matching personal customers by email or phone is processing personal data; document the purpose and keep only the fields you need.
- Give stewards time. A quarantine that nobody works is worse than none, because it hides records from every system.
Summary
Safe matching is layered: exact rules on strong identifiers, fuzzy rules only in combination with exact conditions, quarantine and a person for the doubtful cases, and clear survivorship per field. Boomi DataHub gives you each of those pieces in the model and the stewardship screens. For the wider context, see our Master data management: a practical guide.