Your automation only sees the first 100 records, and nobody notices

A workflow ran clean for six months and still missed half the customers

Here's the one that stuck with me. A client had a Make scenario syncing CRM contacts into their newsletter tool every night. Green on every run for six months. No errors, no alerts. Then someone in marketing noticed a long-standing customer hadn't had an email in ages. We dug in. The scenario had never processed more than 100 contacts in a single run. The CRM held more than 600.

Nobody had built anything wrong. The workflow did exactly what it said on the canvas: ask the API for contacts, get a list, work through it. The catch was in the list. That API returns only the first 100 records per request, and you have to go back and ask for the rest, one page at a time. That second part was never built, because when you test with a dozen contacts, everything looks fine.

This is the pagination trap. It's one of the quietest failures we run into, and it never raises its hand.

Why almost every API hands you data in chunks

An API that dumped fifty thousand records in one response would choke itself and the connection. So nearly every endpoint serves data in pages. You get a block, usually 25, 50, or 100 items, plus some way to fetch the next one. Sometimes that's a page number, sometimes an offset, sometimes a cursor that points at the next block.

Here's what matters. If you take only the first response and ignore the follow-up pages, you get exactly one slice. Everything behind it doesn't exist as far as your workflow is concerned. The API didn't lie. It gave you precisely what you asked for, and you only asked for page one.

Why testing never catches it

The cruel part is the timing. You build with small data: ten contacts, five orders, three tickets. It all fits comfortably on the first page, so it all works. The workflow goes live, and for months the volume stays under the limit. There's no problem to see, because there genuinely isn't one yet.

The bug doesn't show up when you build it. It shows up when you grow. At some point the data crosses the page size, and from then on it gets silently truncated. No red run, no notification. The workflow keeps processing its first page and reports every run as a success. That's why these things run for months before someone stumbles on them, like the customer who quietly stopped getting newsletters.

How to tell if it's happening to you

A few things make us suspicious. If a workflow processes the exact same round number every run, say always 100, that's almost never a coincidence. It's the page size. If the counts between source and destination drift apart, 600 in the CRM and 100 in the newsletter tool, that's a flashing light. And when people say "some records are just missing" with no obvious pattern, pagination is worth checking before you go hunting anywhere else.

How we build it properly

The fix isn't clever, it just has to be deliberate. Instead of one request, you need a loop: grab the first page, process it, ask for the next, and repeat until there's nothing left.

In n8n the "Return All" option on many nodes handles this for you and walks the pages automatically. Where it doesn't exist, you build the loop yourself with an HTTP node and its pagination settings. In Make you reach for an iterator or a repeater and bump the page or offset until the response comes back empty. With Zapier it gets awkward, because the standard triggers usually hand you only the newest items and real loops aren't really part of the design. That's one of the spots where Zapier hits a wall on larger datasets.

The part people get wrong is the stop condition. A loop that can't tell when it's done either runs forever or quits too early. The reliable rule: keep going while the API returns a full page, and stop the moment a page comes back with fewer items than the page size, or empty.

The one check that makes it visible

We always end up recommending the same simple thing, and it takes five minutes. At the end of each run, have the workflow compare two numbers: how many records the source holds, and how many it actually processed. If they don't match, fire an alert.

That comparison catches more than pagination. It catches any kind of silent data loss. It turns a bug that would otherwise hide for months into a message you get the same day. For us it's now standard in any workflow that processes records in bulk.

If you're not sure whether your automations see all your records or just the first page, our free Automations Check is a good place to start. We'll walk your live workflows with you and look hard at the places where data goes missing without anyone noticing.