What to Fix First in a Surveillance Workflow That Keeps Duplicating Case Records

You look at the dashboard and see it: 14,832 case records. You know there are only 9,500 unique people. The duplication has been creeping up for weeks, maybe months. Nobody noticed until the analyst complained that her weekly report showed the same Jane Doe three times with slightly different addresses.

Here's the hard truth: most duplicate case records aren't caused by a single bug. They're the product of a chain—bad dedup thresholds, ingestion races, manual reentry, and data source drift. If you try to fix them all at once, you'll break something else. So you need a triage order. This article gives you one: fix the source first, then the algorithm, then the state machine. Skip the order and you'll be cleaning up duplicates forever.

Where Duplicate Case Records Actually Come From

Public health surveillance and matching errors

Epidemiological case records arrive from labs, hospitals, and contact tracers—each system uses a different identifier. One patient gets logged as 'Jane Doe, DOB 03/14' by one clinic and 'Jane M. Doe, 03/14/1978' by the state lab. The matching logic sees 87% similarity and flags it as a possible duplicate. But that threshold was set by an intern three years ago. The result? Two case records for one infection. I have seen county health departments spend two hours per week manually merging these—time they could have spent on outbreak response. The worst part is not the merge itself; the worst part is the downstream skew. Duplicate records inflate incidence rates, misdirect resources, and ruin weekly situation reports. That sounds fixable until you realize the dedup rules were written for a different disease, a different reporting period, and a different data model. The catch is that nobody updates those rules when a new variant emerges or a new lab joins the network.

Law enforcement case intake and manual reentry

Police departments intake cases via CAD systems, RMS software, and sometimes paper forms that get typed up the next day. An officer writes 'suspicious vehicle, license plate ABC-1234' in the shift log. Another officer, three hours later, files a separate report for the same vehicle—different narrative, same plate. The RMS doesn't cross-reference free-text license plates against open cases. So the system creates two case numbers. We fixed this once by adding a plate lookup widget at the point of entry. But the widget only checked closed cases, not open ones. Why? The original developer assumed open cases would be too volatile to match against. Wrong assumption. Duplicates kept flowing in from the night shift. The real lever here is not better fuzzy matching—it's changing the intake screen so the plate field auto-validates against a cache of recent case records. That takes two afternoons to build. Most teams skip this step because they're chasing a algorithmic dedup solution instead of fixing the input form. That hurts.

Fraud detection pipelines and race conditions

Fraud systems ingest transactions from multiple gateways simultaneously. Two payment attempts for the same suspicious order arrive within 200 milliseconds. One evaluation pipeline marks the order as 'review flagged.' The other pipeline, still running, sees no flag yet—the first write hasn't committed. Both create case records. You now have two identical fraud cases, each assigned to a different analyst. One analyst clears the order; the other blocks it. The order ships, the chargeback hits, and the team spends a week figuring out which case was the 'real' record. The race condition is the root cause, but the deeper problem is that the dedup check is inside the transaction boundary instead of outside it. I have seen teams solve this by adding a distributed lock keyed on the order ID. Simple idea. Execution is harder because the lock itself can become a bottleneck under peak load. Trade-off is clear: accept occasional duplicates during traffic spikes or accept delayed case creation during lock contention. Most teams pick the latter, then forget to monitor the lock timeout. Three months later, duplicates reappear because the timeout was silently lowered during a dependency upgrade. That's how maintenance drift eats your dedup guarantees.

The Dedup Foundations Most Teams Get Wrong

Blocking vs matching: what's the difference?

Most teams throw every field at a similarity function and pray. That's matching without blocking — a computational disaster. Blocking is the cheap gatekeeper: pick one or two high-cardinality fields (case number prefix, date bucket, clinic ID) to carve the record pile into small bins. Only within a bin do you run the expensive fuzzy comparison. I have watched teams skip blocking entirely, run Levenshtein across 800,000 records, and wonder why their pipeline dies at 3 a.m. Wrong order.

The catch is that blocking too narrowly misses duplicates across bins. A patient named 'Jon Smith' on 2024-03-01 and 'Jonathan Smith' on 2024-03-02 land in different date-based bins — your match never fires. Blocking needs slack: overlapping windows or multi-key strategies. Teams treat blocking as a throwaway config, but it's the difference between a pipeline that runs and one that silently bleeds duplicates. That hurts.

Scores, thresholds, and the FPR nightmare

You set a matching threshold at 0.85. Records scoring 0.84 pass through as "unique" — but so do 0.84 pairs that are the same person with a typo. The odd part is—teams tune thresholds by eyeballing ten examples. You get a false positive rate (FPR) you can't see until production explodes. A 1% FPR on 100,000 records means 1,000 ghost duplicates every cycle. Every cycle.

What usually breaks first is the precision-recall trade-off. Push threshold too high: recall drops, duplicates flood in. Push too low: false positives chain-merge unrelated records into one monster case. That's the FPR nightmare — one bad merge poisons downstream analytics for weeks. I have debugged a dashboard where case counts jumped 40% because an 0.79 threshold caught a name collision between two different 'Maria Garcia' entries. Fixed by splitting blocking on mother's maiden name. Simple, but nobody checks the FPR until the seam blows out.

"A threshold that works on last week's data will fail on next week's. The drift is silent, but the duplicates aren't."

— lead data engineer, county health surveillance system

Temporal dedup: why time windows break

Time-window dedup looks elegant on a whiteboard: "If the same person is reported within 72 hours, it's a duplicate." Then a lab runs late — the case comes in at hour 71, gets resolved, and a correction arrives at hour 73. Two separate records, same case. The window snaps shut. Temporal dedup fails because surveillance data doesn't arrive in neat batches; it trickles, retransmits, and corrects itself days later.

Most teams skip this: attach a look-back buffer equal to your longest known reporting lag, not an arbitrary calendar window. Better yet — use event timestamps, not ingestion timestamps. I saw a team using the pipeline clock for dedup windows; a one-hour server delay created 200 duplicate pairs per day. The fix cost three lines of code. The lesson is cheap if you catch it early — expensive if you hit revert on a month of merged records.

Patterns That Actually Keep Duplicates Out

Deterministic blocking with business keys

Most dedup systems start with fuzzy matching. They throw Levenshtein distance at name fields. The result? A lot of near-misses, false positives, and a table that still has three copies of the same outbreak report. I have seen workflows spend 80% of their compute budget comparing strings that should never have touched a comparator. The fix is boring and it works: deterministic blocking on business keys. In surveillance data, a case record almost always carries a government ID — a specimen barcode, a lab accession number, a patient index from the provincial system. You define that field as the block key. Every record with the same key lands in the same bucket. You then compare only within buckets. That reduces the pair space from O(n²) to O(n * block size). The catch is that some records arrive with missing or malformed keys. The pattern demands a secondary block — often date-of-onset plus soundex of surname — for the unkeyed tail. Most teams skip this fallback. Then the tail grows, and duplicates creep back in. Wrong order: you need both blocks from day one.

Reality check: name the epidemiology owner or stop.

Reality check: name the epidemiology owner or stop.

Block first. Match second.

Hash-based matching on canonical fields

Fuzzy matching on raw input is a trap — addresses typed in dialect, names with honorifics, dates in different regional formats. Teams normalise these fields, then compare originals anyway. That hurts. A proven pattern is to compute a SHA-256 hash on a canonical form of the record *before* it enters the matching tier. You strip whitespace, convert all letters to uppercase, reorder address components to a standard layout, and run the concatenated result through a hash. Any incoming record with the same hash is an exact duplicate at the canonical level. No comparison cost. No transposition error. The trade-off is that hash matching misses near-duplicates — a misspelled name that still refers to the same patient. So you use hashes as a pre-filter: rapid reject for exact copies, then deterministic blocking for the rest. What usually breaks first is field selection. Teams include too many columns in the canonical string — a comment field that varies per entry, an investigator ID that changes on re-submission. The hash changes every time. I have watched a team rebuild their entire dedup pipeline three times because they kept chasing hash mismatches caused by a free-text note field. Keep the hash set minimal: stable identifiers, date of birth, sex, and the core event date. Nothing else.

Small hash, big gain. Overstuffing it kills the pattern.

'The moment you hash a free-text field, you're not deduplicating — you're snapshotting noise.'

— platform engineer, after three failed deployments

Idempotent ingestion with replay protection

Duplicate records don't always come from upstream chaos. Sometimes the pipeline itself is the culprit — a Kafka consumer restarts and reprocesses the last batch, an API client retries a POST that actually succeeded, a file lands twice because the SFTP watcher saw a re-upload. The pattern for this is idempotent ingestion: every record carries a unique `message_id` or `event_uuid` generated at the source, and the workflow refuses to insert a record with a `message_id` that already exists in the database. No comparison, no fuzzy logic. Pure dedup at the ingestion edge. The pitfall is that many source systems can't generate stable UUIDs — or they reuse the same ID for different events. You end up dropping legitimate records. The fix is to combine the source identifier (like a lab code) with a monotonically increasing sequence number. That makes the tuple unique per workflow instance. Most teams I have worked with implement this as a UNIQUE constraint on a (source, message_id) composite column. The constraint throws on conflict, and the ingestion handler either skips or logs the duplicate. Replay protection is not a dedup strategy — it's a hygiene layer. Without it, every other pattern you build sits on sand. The odd part is how many teams skip this because they assume data arrives once. It never does.

Anti-Patterns That Make You Hit Revert

Fuzzy matching on everything

I once watched a team turn on Levenshtein distance across every field — name, address, date of birth, even the case ID suffix. Duplicate detection went from 12% to 94%. They deployed on Thursday. By Monday morning the queue held 2,000 false-positive merges. Two sisters with similar names, same zip code, different conditions — merged into one record. The patient lost her medication history. The fix looked genius on a spreadsheet; in production it blew the seam out. Fuzzy matching feels like a safety net. The catch is: it catches everything, including things that should never touch. The trade-off is brutal — recall gains of 20 points often cost 35 points of precision. Most teams hit revert inside 48 hours.

Tighten your fuzzy scope instead. Target only fields where typos actually recur: phone numbers, email aliases, provider IDs. Leave names alone unless you’ve blocked exact duplicates first. Wrong order.

Manual override loops that bypass dedup

Here’s a pattern that looks responsible: a supervisor gets an alert, reviews a possible duplicate, and marks it “confirmed — keep separate.” The system logs the override. Next week the same two records collide again. And again. The manual override loop creates a shadow permission — the human said no, but the automated workflow never learned from that decision. It just retriggers on the next ingestion cycle. That sounds fine until the queue fills with 400 “resolved” items that keep screaming. What usually breaks first is the audit trail: nobody remembers who approved the override, or why, and the default fallback is “automatically merge if confidence > 80%.” Suddenly the manual safeguard becomes the pipeline’s most expensive noise source.

The fix is brutal but clean: once an override is written, the dedup engine must hard-skip that pair for a configurable period — six months, a year, forever. No second guesses. If the pair drifts apart in later records, the skip needs an expiration review, not a silent purge. We fixed this by adding an explicit “don’t re-evaluate before” timestamp. Manual overrides stopped leaking.

Over-narrow time windows for duplicate detection

Three hours. That’s the window a team chose because “most duplicates arrive within the same shift.” The logic seemed tight: if two records for the same person appear inside 180 minutes, flag them; otherwise they’re probably distinct visits. What they missed: the night-shift clerk entered a case at 11:58 PM, the day-shift clerk pulled the same demographics at 12:02 AM — four minutes apart across the midnight boundary, but the window reset at midnight. The duplicates sailed through. Over-narrow windows create edge-case blowholes that look like random failures. The pattern is seductive because it reduces false positives during testing; the pitfall is time-zone drift, batch processing lag, and the simple fact that people re-enter data hours later, not minutes.

“We tuned the window on Monday. By Wednesday, the duplicate count was still climbing — our ‘safe margin’ was actually the widest gap we never measured.”

— Lead data engineer, after reverting a 45-minute window

Set the window wide enough to cover the longest observed re-entry delay in your workflow — often 24 or 48 hours. Then add a buffer for weekends. That feels wasteful until you realize a narrow window is a false-negative factory. I’d rather dedup one extra true positive late than miss a merge that creates a ghost duplicate for six months.

The Long Tail of Maintenance and Drift

Schema drift and field format changes

The dedup logic you shipped last quarter is already aging. Not gracefully either. Data sources shift without notice—a CRM adds a prefix field, a legacy system drops leading zeros from patient IDs, a third-party vendor changes date formats from ISO to epoch timestamps mid-quarter. I have watched teams spend weeks tuning fuzzy matching thresholds, only to wake up one morning to a flood of duplicate case records because a single upstream field silently lengthened from 10 characters to 50. The matching algorithm still fires, but now it scores a different distribution of string similarities. What matched at 0.92 last month scrapes in at 0.78 today. That's not a bug report you want to read at 2 AM.

Flag this for epidemiology: shortcuts cost a day.

Flag this for epidemiology: shortcuts cost a day.

The odd part is—schema drift rarely announces itself. Nobody sends a memo saying "we trimmed whitespace on the source table" or "we merged first and middle name into one column." You catch it only when duplicate counts creep upward or, worse, when a near-perfect match suddenly fails and a second case record spills into the queue. We fixed this at one client by logging the hash of each field's schema version alongside every dedup decision. When recall drops, we can pinpoint which field shifted. Most teams skip this step. They shouldn't.

Monitor field format changes like you monitor server disk space. Run weekly distribution checks on string lengths, null rates, and delimiter patterns. When a field that used to have 95% "YYYY-MM-DD" entries suddenly shows 30% "MM/DD/YYYY", your thresholds need adjustment. Or your pipeline needs a normalization step. Either way, surprise schema shift creates hidden duplicates that bypass your entire workflow.

Thresholds that need recalibration

Dedup thresholds decay. Not because the code rots, but because the data drifts. A threshold of 0.85 might have caught 98% of true duplicates six months ago. Now it catches 91%—and generates a rising tide of false positives that your team manually rejects. The catch is that lowering the threshold to 0.80 recovers recall but floods your review queue with candidate pairs that waste analyst time. Raising it to 0.90 reduces noise but lets through more duplicates.

That sounds fine until you realize no static threshold survives a changing data environment. We scraped the monthly dedup logs for a healthcare surveillance system and found that optimal thresholds shifted by 0.04 to 0.07 over five months—small enough to ignore, large enough to cost hours of manual cleanup. The fix is not a better starting value. It's a process: quarterly recalibration against a labeled holdout set of known duplicates and non-duplicates. Run it, adjust, retest. Skip this and your dedup quality becomes a slow leak—invisible until someone audits the backlog.

'Every threshold is a bet against entropy. The house always wins if you never recalculate.'

— Lead data engineer, public health surveillance platform

Hidden duplicates from data source migrations

Migrations break dedup in ways that feel almost malicious. A hospital system migrates from Epic to Cerner, and the patient identifier format changes from 9-digit MRNs to alphanumeric 12-char codes. The dedup engine, still keying on the old pattern, treats every new record as unique. Result: duplicate case records for every returning patient seen in the first week post-migration. The worst part is that the workflow logs show zero errors—no crashes, no timeouts, just a perfectly clean pipeline producing perfectly wrong output.

Most teams test migrations on throughput, data completeness, and field mapping. They forget to test dedup recall against a pre-migration duplicate sample. The fix is cheap: before go-live, run a batch of 500 known duplicate pairs through the target system's matching logic and measure how many it still catches. We saw a 23% recall drop in one migration that everyone else called "successful." Nobody checked because nobody expected the matching layer to fail silently.

What to do? Add a migration-specific dedup validation step to your cutover checklist. Flag any field that changes length, encoding, or composition—even if the business says "the same data, just different format." It's not the same data. Not to your fuzzy matcher. Not ever.

When NOT to Fix Duplicates at the Workflow Level

Upstream data quality issues you can't fix downstream

Sometimes the duplication isn't a workflow problem at all — it's a firehose problem. I once consulted with a local health department that kept seeing the same case appear three times within four hours. They rebuilt their entire dedup pipeline, added checksums, wrote custom matching logic. Nothing worked. The root cause? Their upstream EHR integration was polling the same patient encounter across three overlapping API endpoints, and none of them used idempotency keys. The workflow was innocent. You can't filter out garbage at the dedup stage if the garbage is structurally identical except for a timestamp that shifts by 47 milliseconds. The fix lives upstream, in the ingestion contract. Stop rebuilding the filter. Block the source.

Here's the rule of thumb: if the duplicate records carry identical clinical data but different system-of-origin IDs, and the duplication pattern follows a fixed cycle — every 7 minutes, every hour on the hour — you're fighting a system integration bug, not a case management flaw. Pouring dedup logic into the workflow for that's like mopping the floor while the pipe is spraying. The real move is to call the data engineering team and enforce a single source-of-truth connector. That feels slower. It's faster by a factor of ten.

Regulatory requirements to keep all records

The other scenario where workflow dedup becomes a liability: the regulator demands every record stays. Some public health surveillance feeds require full event history, including what looks like duplicates. A case may be re-reported by a different lab, or a patient might be retested and registered again under a slightly different name. Scrubbing those duplicates at the workflow level kills auditability. One team I worked with hit exactly this wall — they merged two records that, to their eye, were obvious duplicates. Turned out one was a confirmed positive from a reference lab and the other was a screening test. The reporting agency needed both timestamps, both identifiers, both result values. The merge erased a required chain of custody.

Regulatory double-capture is a real pitfall. The fix is not a better dedup — it's a lineage-preserving merge. You keep both records visible in the surveillance database, link them with a parent-child foreign key, and only collapse the display for operational dashboards. The workflow should tag duplicates as related, not removed. That sounds nuanced. It's. But shipping duplicates into storage is cheaper than explaining to an auditor why a required event vanished. The trade-off here is between clean data and defensible data. If your compliance framework mandates full event retention, choose defensible every time.

'We spent six weeks building a dedup filter. Then the state said every lab report must be preserved individually. We deleted six weeks of work.'

— Senior epidemiologist, state surveillance program

Odd bit about epidemiology: the dull step fails first.

Odd bit about epidemiology: the dull step fails first.

When duplicates are a symptom, not the disease

A third scenario: the duplication rate spikes overnight, and the team scrambles to harden the workflow. The harden works for a week. Then it spikes again. Then again. What usually breaks first is the framing — you keep treating duplicates as the problem, when they're the fever, not the infection. I have seen this pattern in syndromic surveillance feeds where a hospital changed its admission protocol, which caused every pneumonia case to be entered twice: once by the ER triage system and once by the inpatient registration system. The dedup logic got more complex, false-positive merges rose, and patient linkage accuracy dropped. Meanwhile, nobody asked the hospital why case counts suddenly doubled.

The pattern is: one external configuration change upstream triggers a persistent duplicate cascade. The workflow team can't see that change; they only see the messy output. The painful truth is that over-investing in workflow dedup for a drift-driven problem creates a brittle system that breaks harder when the next upstream change lands. The better response is to instrument a duplicate-rate monitor, set a threshold (say, >15% duplicates for more than three days), and pause automated dedup. Investigate upstream. Find the protocol change, the mapping shift, the new data source. Fix that. Then re-enable dedup. Fix the disease, not the temperature.

Before you write another matching rule, ask yourself three questions: Can I trace this duplicate to a single upstream event? Does the regulator require me to keep both copies? Is my dedup rate stable or climbing? If the answer to the third is climbing and the first is yes, step away from the workflow. Go talk to the people who push data into the pipe. That conversation is the only fix that holds.

Open Questions and FAQ on Duplicate Case Records

Should You Ever Allow Duplicates?

Yes—but only when you know exactly why they exist and you trust the downstream consumer to handle them. I have seen teams burn days scrubbing records that absolutely should be duplicates: two separate positive lab results from the same patient on consecutive days, or a manual entry that fixes a scraping error from the original. The fix is a semantic dedup policy, not a blanket block. If your ingestion layer rejects any record with a matching name and date, you lose the second result. That hurts. The trade-off is obvious once you map which fields the business actually uses to distinguish cases. If the downstream dashboard counts unique encounter IDs, duplicates might be invisible noise. If it sums dollar amounts by patient, every duplicate is a dollar error waiting to surface.

Better question: what happens when you let them through?

The odd part is—some surveillance systems depend on partial duplication. Syndromic surveillance, for example, often wants a rolling window of chief complaints, and duplicate drops can obscure a true spike. My rule: allow duplicates only if you tag them with a dedup confidence score and a replay flag. That way you can retroactively purge or collapse them without a full re-ingestion. Most teams skip this; they flip a boolean and walk away. Don't.

How to Handle Timing Windows for Ingestion

The classic pitfall is a stale ingestion window—say, a batch job that runs hourly and re-fetches the last 60 minutes of data. If a record lands in minute 59 and the job starts at minute 60, that record gets pulled. Next hour, the same record is still within the 60-minute window (minute 59 to minute 119), so it gets pulled again. You just created a permanent duplicate. The fix is idempotent dedup keys before the window check—hash the record's source, original ID, and an ingestion timestamp, then store the hash. Any subsequent pull that matches that hash gets dropped, regardless of the window overlap.

What usually breaks first is the source system's own timing.

Some hospitals send lab results 72 hours late. Your dedup key must tolerate late arrivals without collapsing them into an earlier record. We fixed this by using a composite key: source + patient ID + event date, but not ingestion timestamp. That way a late arrival creates a separate record—which you can then reconcile with a nightly merge job. The catch is that merge jobs introduce their own duplicate risk. Set a hard cutoff: any record older than 30 days doesn't merge; it becomes a footnote.

What Monitoring Thresholds Make Sense?

Stop staring at the total duplicate count. Start watching the rate of change in dedup matches per 1,000 records. A spike from 2% to 8% in a single batch usually means a source changed their format or an integration broke. I have seen this three times in production: once because a vendor added a trailing space to patient names, once because a timestamp shifted from UTC to local without notice, and once because a health department changed their record IDs mid-quarter. None of those tripped a volume alert. The dedup-match rate caught all three.

“A 3% duplicate rate is fine. A sudden jump to 3.4% is a fire drill that nobody answers.”

— senior data engineer at a state syndromic surveillance program

Set a hard threshold for over-deduplication—when your matcher starts throwing away more than 1% of incoming records as false positives. That usually means your fuzzy-match parameters are too aggressive or your window is too wide. The fix is a manual review queue: send flagged records to a human for 48 hours, then adjust the threshold. Most teams skip this because it feels like overhead, but the alternative is silent data loss that compounds for months.

Track one more thing: the time-to-fix for a confirmed duplicate outbreak. If it takes your team longer than two hours to identify the root cause and push a config change, your monitoring is too slow. Drop a Slack alert on a dedup-match rate change above 2 standard deviations from the 7-day rolling mean. Test it weekly with a synthetic duplicate injection. Not yet doing that? Start next Monday. That's the single highest-leverage action you can take—everything else is cleanup.