A store I work on imported about 2,900 new titles on Friday night. By Saturday afternoon, 862 had pictures. The rest showed a gray placeholder, and the owner reasonably wondered whether anything was happening at all.

The fix for why the pictures had failed was ready: each picture job now does one download, with a hard time limit, so the host can’t kill it halfway. I installed it with a backup first. Then I looked at the queue, expecting a few hundred jobs, and found 2,017 waiting. They’d been waiting since Friday.

WordPress has a background queue for jobs like these. On a normal server something runs it every minute. On this host it only runs when a person loads a page, so the queue moves when the store has visitors and sits still when it doesn’t. That’s slow, but it isn’t stuck. This queue was stuck.

The reason was 108 old claims. Before a worker starts a batch of jobs, it writes a claim so two workers don’t grab the same ones. When the host killed those workers at its 30-second limit, they died without letting go. The queue has a limit on how many batches can be claimed at once, and 108 dead claims were well past it. Every new worker looked at the table, decided too many batches were already running, and left. Nothing was running. The queue was full of ghosts.

I cleared the claims that weren’t attached to any live job and started three workers from the server’s command line, where the 30-second limit doesn’t apply. The queue went from about 5 pictures every 5 minutes to about 9 a minute. By 8 PM all 2,860 were in.

Then I made my own mistake. My first worker had a 55-second timeout of its own, and I hadn’t thought about what that does to a job in the middle of a download. It does the same thing the host was doing. Seven pictures got cut off that way before I noticed and took the timeout out. I queued them again at the end, and they finished.

The part I keep thinking about is how long the queue looked fine. Nothing in the store’s admin said “blocked.” The jobs said pending, which they were. You had to count the claims to see it. A system that’s stopped and a system that’s waiting for its turn can look exactly the same from where the owner stands, and only one of them gets there by morning.