Why BullMQ throws 'Custom Id cannot contain :' — and 1,034 green tests missed it

On the morning of 2026-08-24, an upload route in the RS TROPHY admin API — the platform I run as RUAMSUK PLATING LIMITED PARTNERSHIP at RUAMSUK PLATING LIMITED PARTNERSHIP — returned a 500 with nothing in the log. The route saved a photo template's background plate, then queued a preview render of it. I was working through the social auto-posting module with an AI coding agent, as I did for most of that month, and isolating the enqueue on its own finally produced the real message: Custom Id cannot contain :.

That string comes from BullMQ. A custom jobId is how you make a job idempotent — enqueue the same id twice and the queue keeps one — and every stable id in the social module was built the same way: publish:${postId}, sample:${templateId}, generate:${ruleId}, clean:${templateId}. BullMQ refuses all four at queue.add. Scheduled publishing, "Generate now", the template sample render and the AI plate clean could not enqueue a single job between them.

//What BullMQ actually refuses

The BullMQ docs on job ids say a custom id must not contain :, because it is the separator in BullMQ's Redis keys, and suggest - or _ instead. The thrown error is newer than the advice. It arrived in a pull request merged in September 2025, fixing an issue filed by a stranger whose flow children came back undefined because their ids held colons. I read the check itself in the version our lockfile pinned, bullmq 6.0.9 — job.js:912 in the installed package — and ran the old ids through it:

const rejected = (id: string) =>
	id.includes(':') && id.split(':').length !== 3

rejected('publish:507f1f77')     // true  → throws 'Custom Id cannot contain :'
rejected('clean:507f1f77')       // true  → throws
rejected('restage:507f1f77:7')   // false → exactly two colons: let through
rejected('publish-507f1f77')     // false

Read the predicate closely and the commit message I merged that morning is wrong in one place. It said every custom job id was illegal. Exactly two colons pass — a compatibility branch for old repeatable jobs, which BullMQ's own comment marks for replacement in its next breaking change. Photo Studio's restage:${batchId}:${index} had exactly two. So four of five builders were broken, not five, and photo restaging was never blocked by this. It was hyphenated anyway: leaning on a branch scheduled for deletion is the same bug on a slower timer.

//Why 1,034 green tests could not see it

The queue had tests. social-sample-queue.test.ts replaced socialQueue.add with a spy, enqueued twice, and asserted the job id it was handed:

const add = makeSpy(socialQueue, 'add')
add.mockImplementation(() => Promise.resolve({} as never))

await enqueueSampleRender(ID)
await enqueueSampleRender(ID)

const opts = add.mock.calls[0]?.[2] as { jobId?: string }
expect(opts.jobId).toBe(sampleJobId(ID)) // 'sample:507f…' — fine by the spy

That test is right about what it checks. It proves the id is stable: the same template gives the same id, so a second edit replaces a pending render instead of paying for two. It says nothing about whether the id is legal, because the validation lives inside BullMQ, behind queue.add — exactly the call the spy replaced. The mock was more permissive than the library it stood in for, and a total failure fit through that gap.

The 1,034 is not a rhetorical number. It is the suite size quoted in the follow-up commit, and re-running a snapshot of the merge that shipped generate:${ruleId} on the night of 2026-08-23 gives exactly 1,034 passing tests and zero failures. The commit just before the fix ran 1,066, also green. A bigger suite of the same kind of test buys nothing here — every one of them was a spy.

//A guard that reads the source, and is proven able to fail

The fix, cb0b9eb9, swapped each colon for a hyphen and added queue-job-ids.test.ts, which asserts the property BullMQ enforces on the string itself — no Redis needed. Its first version was a hand-written list of the five builders that existed that day, which is the same blind spot moved somewhere new: a sixth builder in some future module is not on the list, and nothing notices. The follow-up, 92a239ed, made the source the input.

// every JobId builder literal and inline jobId: '…' under apps/api/src
const literals = idLiteralsInSource()

test('the scan actually finds the ids it is supposed to be checking', () => {
	const found = literals.map(l => l.literal)
	expect(found).toContain('publish-')
	expect(found).toContain('football-bootstrap-squads')
	expect(literals.length).toBeGreaterThanOrEqual(12)
})

test('not one of them would be rejected by BullMQ', () => {
	expect(literals.filter(l => bullmqRejects(l.literal))).toEqual([])
})

The first test is the one I would keep if I could keep only one. A regex that silently matches nothing passes forever while proving nothing — the same false confidence that let the original bug ship. If a refactor changes how ids are written, the scan finds fewer, this fails, and the pattern gets updated. The commit also closed a quieter route in: the football ingester builds a resume id by interpolating a job-name constant, so a colon could reach that id without ever appearing in a literal. The constant's values are asserted too.

Then it was proven able to fail rather than trusted because it was green. Reintroducing clean:${templateId} fails four assertions in that file, the source scan among them; re-run on a snapshot while writing this, the count holds at four of eight. Putting restage back to its two-colon form fails exactly one — the strict no-colon test — because the scan checks BullMQ's real predicate, and the real predicate lets it through. That single failure is the reason the file holds a stricter line than the library does.

//The docs were teaching the bug

The part I think about most is not the code. cb0b9eb9 hyphenated the builders but left their descriptions alone, and 6f9ed9a5 found the old format still written as house convention in eight lines across seven files: source comments, the shared package, the social README, and two files under .claude/skills/api-conventions/. Those last two are what my coding agent auto-loads before it writes API queue code. They documented, as the rule to follow, the exact string queue.add throws on.

Most of this module was written with that agent, and on ids it was doing what it was told. A bad convention in an agent's instructions is not a typo in a README; it is a generator. The test catches a recurrence at the gate, but the docs are what stop it being written in the first place, and until that afternoon they argued for it. The rule now sits in the section where a new queue gets designed — separate custom ids with -, never : — with the failure mode and the bullmq line number attached.

A convention that only says what to type gets copied; one that says what breaks gets understood.

— commit 6f9ed9a5

//What it cost, and what I got wrong

• "Generate now" was broken by its own fix. The night before, I moved it into a background job because the inline request ran so long that a 502 came back after ~96 s, over a generation that had actually succeeded (part 4 covers that day). The new job's id was generate:${ruleId}, so from that merge until the fix the button could not enqueue at all — a misleading 502 traded for a guaranteed error, for about twelve and a half hours on main.
• The publish path was the oldest. publish:${postId} dates from the publish queue's first commit on 2026-06-22, and BullMQ's check was already in the release the lockfile pinned that day. By the time of the fix, scheduling, retrying, "post now" and the boot-time catch-up all went through that one enqueue — nine weeks of a path that could not have queued one job. In the fix commit's own words, it was safe to ship without a migration precisely because the feature never worked.
• The commit message overstated it. "Every custom job id was illegal" was four of five. I would rather correct that here than leave the commit log as the record.
• The guard has upkeep. Its floor of 12 had to drop to 11 the same afternoon, when the AI plate clean was deleted (part 6) and took its id builder with it. A count floor is a number every deletion has to remember to edit.
• It is still a regex. It sees two shapes: a JobId builder with a template literal, and an inline jobId: string. An id built some third way is invisible to it, and the self-check only proves the scan still finds the ids it already knew about.

The honest alternative was an integration test that calls the real queue.add against a real Redis. It would have caught this on its first run, along with failures a string predicate never will. I chose the string because BullMQ's rule is a property of the string, and a test that needs no infrastructure runs on every save — but the cheap test is not the complete one, and I would not claim it is.