The 2am Test: Matching Data Movement to the Team That Must Run It

Architecture diagrams are optimistic. They show clean arrows between systems and assume that data arrives where it is needed, when it is needed, without anyone having to think about it at the weekend. Operational reality is less tidy. A payroll file fails at 1am. A change data capture connector stops after a database upgrade. An event consumer falls behind and nobody notices until a customer record is wrong in three systems.

The choice between batch, change data capture and events is therefore not just a latency decision. It is a decision about who will operate the pattern, how failures will be diagnosed, and whether the organisation can recover before the business notices. The right pattern is the one the team can debug at 2am, not necessarily the one that looks most modern on a slide.

Three patterns, three operational bills

Batch movement runs on a schedule. Data is extracted, transformed and loaded in files or controlled jobs. It is old-fashioned in the best sense: predictable, auditable and easy to rerun if the job is designed properly.

Change data capture, or CDC, reads database change logs and turns inserts, updates and deletes into a stream of changes. It narrows the latency gap and avoids full extracts, but it introduces dependencies on database internals, connector configuration and schema evolution.

Event streams publish domain events to a broker or streaming platform. Consumers subscribe and react. This is the most flexible pattern and the most demanding. It requires event schemas, consumer ownership, idempotency and a replay strategy.

Each pattern has a place. The mistake is choosing by aspiration rather than by operational fit.

Batch: reliable, late, and quietly critical

Batch is often treated as the legacy option, yet it remains the correct choice for many UK business processes. Payroll is the obvious example. A monthly payroll run, PAYE reporting and BACS payment files depend on controlled totals, sign-off and a clear audit trail. The latency requirement is measured in hours or days, not seconds. The volume can be large, but it is predictable. Failure tolerance is low at the deadline, which is exactly why batch control matters.

A well-run batch process has checksums, record counts, control totals and an exception report. It has an owner who knows how to rerun a failed job without duplicating payments. It has a runbook that describes what to do when the source file is late or the target database rejects a row. The 2am test is passable because the failure modes are familiar.

Batch becomes dangerous when it is invisible. An overnight reconciliation job that quietly becomes business-critical is a common problem. It starts as a reporting convenience and ends up feeding a regulatory return. If nobody monitors the job chain, a silent failure can leave finance teams reconciling by hand for days.

CDC: precision with sharp edges

CDC is attractive when a system needs near-real-time copies of another system's data without polling the source. Customer record synchronisation is a typical case. A CRM update should appear in a data warehouse or operational store within minutes, and a full extract every hour is wasteful.

CDC handles volume well, but it is sensitive to change. A database schema change can break a connector. A new column may be ignored or may cause a type mismatch downstream. Deletes and updates need careful handling, because a consumer that only sees inserts will slowly drift from the source. Ordering matters, and replay is not always as simple as pressing a button.

The operational bill includes connector ownership, lag monitoring, schema registry discipline and a documented replay procedure. The 2am question is direct: can the on-call engineer identify whether the problem is in the source database, the connector, the broker or the target? If the answer is no, CDC is not yet a safe choice.

Events: power and prerequisites

Event streams are the right pattern when several systems need to react to the same business fact. A customer changes address, an order is cancelled, a payment is received. Publishing a domain event allows multiple consumers to respond without the producer knowing about each one.

That decoupling is real, but it is not free. Events require governance. Who owns the event schema? What happens when a field is deprecated? How are duplicate deliveries handled? What is the replay strategy for a consumer that was offline for six hours? Exactly-once delivery is not a feature you can assume; idempotency is a design requirement you have to implement.

Events also demand skills. A team that can write SQL and schedule jobs may not have experience with consumer groups, partitioning, dead-letter queues and schema compatibility. That is not a reason to avoid events forever. It is a reason to be honest about the training and support required before making them critical to payroll or finance.

A decision framework that starts with operations

Four questions help narrow the choice.

Latency need. If the business can wait hours, batch deserves serious consideration. If it needs minutes, CDC or events may fit. If it needs seconds, events are more likely, but only if the operational capability exists.

Volume and shape. Large, predictable extracts suit batch. Continuous changes suit CDC. Multiple consumers reacting to the same business fact suit events.

Failure tolerance. If a failure can be corrected by rerunning a controlled job, batch is manageable. If consumers must replay a stream without duplicating effects, events require stronger engineering discipline.

In-house capability. This is the criterion most often skipped. A pattern that the team cannot monitor, replay and repair is a liability, however elegant it looks.

Scenarios that make the choice concrete

For a monthly payroll run, batch is usually the right answer. The process needs control totals, approval gates and a clear rerun path. An event stream would add complexity without removing the need for reconciliation.

For finance reconciliation, batch also has a strong claim. Bank feeds, ledger extracts and control totals are naturally bounded. CDC can help if the ledger must be reflected near-real-time in a reporting store, but the reconciliation itself still needs a batch-style control framework.

For customer record synchronisation, the answer depends on the number of consumers. A single CRM-to-warehouse feed may be well served by CDC. If several systems need to react to a customer change, events become more attractive, provided the schema and replay model are owned.

The 2am test as a design gate

Before committing to a pattern, ask the operational question. When this fails at 2am, who is called? What alert fires? What does the runbook say? Can the team replay or rerun without duplicating data? How will they know when the backlog is clear? If the answers are vague, the pattern is not ready for a critical process.

The most robust architecture is not the one with the most arrows. It is the one whose failure modes are understood, whose ownership is clear and whose recovery path has been rehearsed. For UK organisations moving customer, finance and payroll data, that is a more useful standard than novelty.