Why every serious automation needs a queue

Your automation works fine until the day a thousand things happen at once

Here is a pattern I have watched play out more times than I can count. A business builds an automation. It runs for months without a hiccup. Everyone forgets it exists, which is the highest compliment an automation can get. Then one day something spikes, a sale, a campaign, a mention that goes further than expected, and the whole thing falls apart in a way nobody saw coming. Orders vanish. Invoices get created twice. Customers wait hours for a confirmation that used to arrive instantly.

The reflex in that moment is to blame the tool. It is almost never the tool. The problem is that the automation was built to do all its work the instant an event arrives, and doing all your work the instant an event arrives is a promise you cannot keep once events stop arriving one at a time. What is missing is a queue.

I want to walk through what a queue actually is, when you genuinely need one, what the popular no-code platforms can and cannot do here, and three real ways I have built this. If you run anything that processes orders, payments, signups, or leads, this is worth twenty minutes of your attention.

What a queue is, in plain terms

Most automations are synchronous. An event comes in, and the same flow that receives it also does the work: write to the CRM, create the invoice, send the email, update inventory. Receiving and processing are welded together. That is fine as long as events show up one by one with room to breathe.

A queue splits those two jobs apart. At the front sits something whose only task is to accept an event and put it somewhere safe. That is fast, because it barely does anything. At the back sits a second process that pulls events out of that safe place and works through them at its own pace. In between sits the buffer, which is the queue itself. It holds whatever has not been handled yet.

The two sides have names. The producer creates work and drops it off. The consumer picks work up and finishes it. The whole trick is that they run independently. A thousand events in twenty minutes? The producer accepts them in seconds and parks them. The consumer then works through them at whatever speed the slowest downstream API allows. Nothing is dropped, nothing gets overrun.

Fast is not the same as instant

The most expensive misconception in process automation is treating fast and instant as the same word. Your customer expects their order to register right away. Your customer does not expect the invoice to appear in your accounting system in that same second, and does not care. You are allowed to separate those two expectations, and a queue is how you do it.

Accepting safely means the event is captured, it cannot be lost, and the sender gets an immediate acknowledgment. Whether the real work happens two seconds or two minutes later is invisible. A lost order, on the other hand, is extremely visible. Once you internalize that gap, you start building automations differently.

How to tell you actually need one

People ask me this in first conversations: how do I know I need a queue rather than just a tidier automation? The honest answer is that there are three situations where synchronous flows reliably break, and if your business hits any of them, a queue stops being over-engineering and becomes the cheapest insurance you can buy.

Spikes

The obvious one. A promotion, a trade show, a viral moment, a month-end close, holiday traffic. Events do not arrive evenly, they arrive in waves. A synchronous flow processes a wave only as fast as the weakest link in the chain, and the weakest link is almost always a third-party API with a rate limit. A queue absorbs the entire wave immediately and lets the consumer drain it at a rate the API tolerates.

A downstream system that is slow or down

Every system you do not run yourself will be unreachable at some point. Maintenance window, outage, an API that returns errors for forty minutes. In a synchronous flow that means everything happening during those forty minutes fails outright. With a queue it means events pile up patiently in the buffer, and the moment the system returns, the consumer works through the backlog. The outage becomes a delay instead of data loss. That is an enormous difference, and it is decided entirely by this one architectural choice.

Order matters

Sometimes two events touching the same record must not be processed at the same time. A customer updates their address and cancels one second later. Run both flows in parallel and the faster one wins, and which one is faster comes down to luck. A queue can hold events in the order they arrived and process them one after another. An entire class of bugs, the kind that shows up only occasionally and is therefore nearly impossible to reproduce, simply disappears.

If your logs show clusters of failures at predictable busy times, you are not looking at a tooling problem. You are looking at an architecture problem, and the fix has a shape.

What the no-code platforms really do here

Now the part where I make no friends. The popular no-code platforms love to advertise reliability and error handling, but a real queue worth the name is something most of them do not ship. You have to look closely at what each one actually means.

Make and Zapier

Make has an internal processing order and can push failed runs into a retry mode. Zapier has autoreplay and, on some plans, a delay step. This sounds like a queue and does not behave like one. These mechanisms are built for the occasional single failure, not for the wave. Send a thousand events at once and both platforms keep firing as many runs as they can against the downstream system, because their core model is to handle every trigger as immediately as possible. The decoupling that a queue is all about never happens.

You can improvise. A common trick is to write incoming events into a table or datastore first, then have a second scheduled flow process the new rows every minute. That genuinely is a homemade queue, and for plenty of businesses it is entirely enough. You just want to know that you are building and maintaining it yourself rather than buying it.

n8n queue mode

n8n is the one big platform with a real queue mode built in. Workflows no longer run in the same process that triggers them, they get distributed to multiple workers through a Redis queue. That is the producer-consumer model without having to reinvent it by hand. If you self-host n8n and process serious load, use it. Self-hosting has its own costs, which I have written about separately, so go in with eyes open.

The three words that decide the project

The moment you add a queue, three terms show up that you need to understand before you go live. Skip them and you trade your old problems for new ones that are harder to find.

At-least-once means build it idempotent

Almost every practical queue guarantees at-least-once delivery, not exactly-once. Occasionally the same job gets pulled twice, for example when a consumer crashes mid-work and the job is released again after a timeout. So the processing flow has to survive a duplicate delivery without doing harm. An invoice must not be created twice just because its trigger arrived twice.

The cure is idempotency, which I have written a whole piece about. In short, the consumer checks before acting whether this specific job has already been done, usually via a unique event identifier, and silently skips it if so. Add a queue and forget idempotency, and you have traded lost records for duplicate ones, and duplicates are often the more expensive mistake.

A dead letter queue for what keeps failing

Some jobs cannot be processed no matter how many times you try. A corrupt record, a character the target API rejects, an order for a product that no longer exists. If the consumer keeps pulling that item, failing, and putting it back, you get an infinite loop that jams the whole queue. After a set number of attempts, usually three to five, the job should move not back into the main queue but into a separate holding area, the dead letter queue. There it sits safely, blocks nothing, and a human can look at it later. A queue without this catch-all is a house of cards that collapses on the first bad record.

Visibility, or the queue nobody watches

The third one is less technical and more important than most people expect. A queue hides delay, that is its job, and that is exactly what makes it treacherous when nobody is watching. If the consumer works slower than the producer feeds it, the buffer grows, quietly, until it holds hours of backlog. You need one number on a dashboard: how many jobs are waiting right now. If that number climbs past a threshold and stays there, something is broken. Without it, you learn about the backlog when a customer calls.

Three architectures I have actually built

Theory only takes you so far. Here are three implementations from real projects, at different sizes, with different means.

A buffer in front of the CRM for a B2B retailer

A retailer of about forty people ran every purchase through one big flow: receive order, write to CRM, create invoice, update stock, send confirmation. On a normal day, no problem. On a discount Friday, over a thousand orders landed in the first twenty minutes, the accounting API hit its rate limit around the three hundredth call, and things went sideways: some orders never reached the CRM, some landed twice, some confirmations went out on Saturday afternoon.

We split the one flow into two. The first receives the order, writes it to a simple table with a status field, and acknowledges the shop immediately. The second runs every thirty seconds, grabs the next twenty open rows, and processes them. If a step fails, the row stays open and is retried next run, up to five times, then flagged as a problem. No new infrastructure, all built with the platform already in use. The next discount Friday, everything was processed by midnight, clean, no duplicates, no losses. The customer never noticed, which was the point.

A Redis list in front of an invoicing API for a solo business

A solo operator selling online courses had the same problem at a smaller scale. Every course launch brought hundreds of purchases in minutes, and the invoicing API allowed only a few calls per second. A table buffer would have worked, but he already ran a small server with Redis on it, so we used a Redis list as the queue. The purchase webhook pushes the order onto the list, a small script pulls items off at the allowed rate and calls the API. The lesson: you do not have to be big to need a real queue. The solo business had the same spike as an enterprise, just smaller in absolute numbers. The architecture was identical.

A managed queue in an enterprise

The third case, a larger company with its own IT department, added a requirement the others did not have: auditability and operational assurance had to be demonstrable. A homemade table queue would have worked technically, but nobody accountable for operations would sign off on it. So we used a managed service from their existing cloud provider, with a built-in dead letter queue, delivery guarantees, and metrics that plugged into their existing monitoring. More effort, and more control and provability in return.

Same core idea, three implementations, staggered by requirement. From a table buffer to a Redis list to a managed queue is not a leap, it is a staircase.

When you do not need one

As much as I believe in queues, let me be just as clear: most automations do not need one. One event an hour, no spikes, an occasional harmless retry, and a queue is surplus machinery you have to maintain for nothing in return. I have written elsewhere about the rule of doing a task by hand once before automating it, and the same spirit applies here. Build the queue when the synchronous flow demonstrably hits its limit, not before.

The signal is concrete and not hard to spot. Your logs show failed runs tied to rate limits, timeouts, or overrun systems, and those failures cluster at particular times. If that happens once or twice a year and does no damage, live with it. If it happens every month-end and costs you rework or customer trust, the moment has arrived.

What I took away from this

Most of the expensive automation failures I have investigated were not about the wrong tool. They came from a single design decision nobody consciously made: receiving and processing were thrown into one pot, because early on, at ten events a day, it made no difference. The difference shows up later, with growth, with the first promotion, with the first post that travels further than expected. And it shows up on a Friday evening when nobody is looking.

So on any automation that handles something that matters, I now ask one question early: what happens if a thousand times the usual volume shows up at once? If the honest answer is that things would get lost or duplicated, a queue belongs in the middle. Not because it sounds sophisticated, but because it is the difference between a delay and a loss. Nobody remembers a delay by Monday. Everybody remembers a loss.