How to never pay a generative API twice: a durable task table, not a retrying job queue

Short answer: the moment you might have been charged is not the provider’s response, it is the instant before the request leaves your process. Write that instant to a database row before the HTTP call, and a worker that dies mid-request becomes recoverable without guessing: nothing on the row means nobody paid and the task can go back to the queue; a saved request id means poll and never resubmit; the boundary written with no id means a third terminal state that no automation is allowed to retry.

TL;DR

• Fix: own task table with a lease, and a submit_started_at column written before the paid call. A queue retries by design; pg-boss says so in its own docs: “the job can run twice and handlers should be idempotent.”
• Alternative: a provider-side idempotency key, if your provider has one. Stripe does. The OpenAI and Anthropic clients ship maxRetries: 2 and send no idempotency header at all.
• Trap: treating an expired deadline as a failure. A poll that ran out of patience is “I don’t know”, and the difference costs money every time you get it wrong.

Why a job queue is the wrong unit of recovery for a paid call

Because retrying is the entire point of a job queue, and for a paid POST a retry is a second charge. I started by running generation steps as ordinary jobs on the queue the rest of the system already used, pg-boss, because that is where every other background task lived. That lasted until I read the queue’s own documentation on what happens when a worker looks dead:

A worker that only looked dead (a network partition, a stalled event loop) keeps running its handler after its job is failed this way, so the job can run twice and handlers should be idempotent.

That is the pg-boss queue reference, and it is being honest rather than apologetic. Every durable queue has the same sentence somewhere, because a queue cannot tell a dead worker from a slow one. The contract it offers is at-least-once, and it tells you to make the handler idempotent. For sending an email, fine. For a handler whose first line spends money at an external provider, “make it idempotent” is the hard part being handed back to you.

The defaults make it concrete. retryLimit is 2, so a failing job runs three times. expireInSeconds defaults to 15 minutes, after which an active job is retried or failed regardless of what the worker is doing. Image generation lands in tens of seconds and video generation in minutes, so a long clip can sail past the expiration while it is perfectly healthy, get failed by the monitor, and get retried into a second charge. You can raise the expiration to 24 hours, the documented maximum, but then a genuinely dead worker sits undetected for a day.

There is a second problem that is not about money. A job that spends twenty minutes polling a remote id occupies a worker slot for twenty minutes. Concurrency on the queue is shared with everything else in the system, and a handful of slow generations will starve the short jobs that the queue exists for.

The unit of recovery had to stop being a job and become a row.

What RFC 9110 actually says about retrying a POST

It says do not, unless you can prove one of two things, and the second one is the design brief for the whole task table. RFC 9110, section 9.2.2:

A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied.

Two escape hatches. The first one, knowing the semantics are idempotent, is what a provider gives you with an idempotency key. Most generative APIs do not give you one, which I will get to. The second, “some means to detect that the original request was never applied,” is not a property of the protocol at all. It is a property of what you wrote down locally before you made the call.

The RFC also has a line about clients that skip both:

Some clients take a riskier approach and attempt to guess when an automatic retry is possible. For example, a client might automatically retry a POST request if the underlying transport connection closed before any part of a response is received, particularly if an idle persistent connection was used.

That paragraph describes the official clients of the largest paid APIs in the world, which is the subject of the next section.

Do the generative APIs give you an idempotency key?

Mostly no, and the clients retry your POSTs anyway. I went looking for this expecting a boring answer and got a much sharper one by reading the SDK sources instead of the marketing pages.

Stripe is the benchmark for how this is supposed to work, and its wording is worth quoting because it names the property that matters:

Stripe’s idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors.

That is the Stripe API reference on idempotent requests. Note the retention rule in the same page: keys can be pruned once they are at least 24 hours old, and a key reused after pruning generates a new request. Even the gold standard has an expiry after which your safety net is gone, which is a good reminder that the client side still has to keep its own record.

The OpenAI side is where it gets strange. The OpenAI Node client carries the full apparatus for idempotency keys. It declares the field, it has a key generator whose default value announces where the machinery came from, and it builds the header:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

protected idempotencyHeader?: string;

protected defaultIdempotencyKey(): string {
  return `stainless-node-retry-${uuid4()}`;
}

// ...later, while assembling headers:
let idempotencyHeaders: HeadersLike = {};
if (this.idempotencyHeader && method !== 'get') {
  if (!options.idempotencyKey) options.idempotencyKey = this.defaultIdempotencyKey();
  idempotencyHeaders[this.idempotencyHeader] = options.idempotencyKey;
}

idempotencyHeader is declared and never assigned anywhere in the file. The condition is therefore always false and the header is never sent. RequestOptions even exposes idempotencyKey with the comment “A unique key for this request to enable idempotency”, and setting it accomplishes nothing, because there is no header name to put it in. I spent a while convinced I had missed an assignment in a subclass. There is none.

Meanwhile the retry predicate in the same file does not look at the method at all:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

private async shouldRetry(response: Response): Promise<boolean> {
  const shouldRetryHeader = response.headers.get('x-should-retry');
  if (shouldRetryHeader === 'true') return true;
  if (shouldRetryHeader === 'false') return false;

  if (response.status === 408) return true;   // request timeouts
  if (response.status === 409) return true;   // lock timeouts
  if (response.status === 429) return true;   // rate limits
  if (response.status >= 500) return true;    // internal errors
  return false;
}

DEFAULT_TIMEOUT in that client is 600000, ten minutes, and a timeout goes down the same retry path. The Anthropic TypeScript client has a byte-for-byte equivalent predicate, the same maxRetries: 2 and the same ten-minute default, and zero occurrences of the string idempot in the entire file.

The default behaviour of both clients, then: send a paid POST, and if the server returns 500 or the socket goes quiet for ten minutes, send it again, up to two more times, with no way for the server to recognise it as the same request. Whether the first attempt was billed is between you and the provider’s meter. The client has no way to ask, and neither do you.

This is not a scandal. It is the reasonable default for an API where most calls are cheap and a dropped connection is more likely than a silently successful one. It stops being reasonable the moment a single call costs real money and the user can see the balance.

The boundary is before the request, not after the response

The instant that matters is the last moment before the send, not the first moment after the answer. This is the one idea the whole design rests on, and it took me an embarrassing amount of staring at a state machine to see it.

I had been drawing the boundary in the obvious place: the task is safe until the provider answers, and becomes committed when it answers. That is wrong, and the reason is that the answer is exactly the thing you do not have when the worker dies. A crash between “socket write completed” and “response parsed” leaves no evidence at all on the client side, and it is the most likely crash window there is, because that window is where all the waiting happens.

Flip it. The task is safe until you begin the request, and becomes unknown the instant you do. That boundary is knowable in advance, so it can be written down in advance:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

async function runImageTask(task: Task, ports: Ports): Promise<void> {
  if (task.status === "SUBMITTED" && task.remoteRequestId) {
    return pollUntilDone(task, ports);       // already paid for, never resubmit
  }

  await ports.repo.markSubmitStarted(task.id);            // the boundary
  const { requestId } = await ports.provider.submit(task.input);
  await ports.repo.recordSubmitted(task.id, requestId);   // now it is addressable

  return pollUntilDone(task, ports);
}

Three writes, and every crash window between them maps to exactly one recovery action. markSubmitStarted is a separate round trip to the database before a call that is about to take thirty seconds, so the cost of the extra write is not worth discussing.

The one rule this imposes on the rest of the code is that nothing may sit between markSubmitStarted and the actual send. No input validation, no prompt assembly, no lazy credential fetch. Anything that can fail there produces a task parked in the unknown state for no reason, and parking a task costs a human’s attention later. Build the request fully, then mark, then send.

Three recovery branches after an expired lease

A supervisor picking up a task whose lease has expired has exactly three cases, and the row tells it which one without any guessing:

In code it is short enough to fit in a screenshot, which is the point:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

type Recovery =
  | { action: "requeue" }
  | { action: "poll" }
  | { action: "park"; status: "OUTCOME_UNKNOWN" };

function recoveryFor(task: Task): Recovery {
  if (task.status === "SUBMITTED" && task.remoteRequestId) return { action: "poll" };
  if (task.submitStartedAt === null) return { action: "requeue" };
  return { action: "park", status: "OUTCOME_UNKNOWN" };
}

Order matters in that function and it is not the order I wrote first. My first version checked submitStartedAt before the request id, which sends an already-submitted task back through the submit path whenever both columns are set, which is always for a healthy submitted task. The request id is the strongest evidence on the row, so it gets tested first.

Why OUTCOME_UNKNOWN has to be terminal

Because every automatic way out of it is a bet placed with someone else’s money. This was the design decision I argued with myself about longest, since a state that requires a human is an operational cost forever.

The two tempting exits both fail. Retrying assumes the request never landed, and if it did land you have paid twice and the user sees one result. Failing assumes the request never landed too, just with a nicer label: the task disappears from the pipeline, the charge does not disappear from the invoice, and the reconciliation happens weeks later when someone asks why the numbers do not match. Both are the same guess wearing different clothes.

The state is therefore terminal, the UI says the outcome could not be determined and suggests checking the provider’s dashboard, and a human presses the button. In exchange, the system never spends money on its own initiative. For a paid API that is the correct side of the error to be on, and the volume is low enough that it stays correct: the branch only fires when a process dies inside the send window, which is a few seconds out of a generation that takes minutes.

The same logic applies to deadlines, and this is where I got it wrong first. Put a cap on polling, because a remote id that never resolves cannot hold a slot forever. My original code marked the task FAILED when the cap was reached. That is the same lie as before: the provider might still be working, or might have finished and billed while my poll loop gave up. An exhausted deadline is not a failure, it is an unknown outcome that happens to be slow. It goes to the same terminal state, with a different reason string.

Claiming work with FOR UPDATE SKIP LOCKED

The claim is a single statement, and PostgreSQL documents this exact use case. From the SELECT reference:

With SKIP LOCKED, any selected rows that cannot be immediately locked are skipped. Skipping locked rows provides an inconsistent view of the data, so this is not suitable for general purpose work, but can be used to avoid lock contention with multiple consumers accessing a queue-like table.

“Inconsistent view of the data” is the point, not a caveat. Two supervisors claiming work must not see the same row, and neither needs an accurate count of what is pending.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

update generation_task t
   set lease_owner      = $1,
       lease_expires_at = now() + $2::interval,
       status = case when t.status = 'PENDING' then 'RUNNING' else t.status end
 where t.id in (
   select id
     from generation_task
    where kind = $3
      and (status = 'PENDING'
           or (status in ('RUNNING', 'SUBMITTED') and lease_expires_at < now()))
    order by created_at
    limit $4
      for update skip locked
 )
returning t.*;

The case matters, and this is the part I got wrong. An earlier version of this statement set status = 'RUNNING' unconditionally, which erases the one signal that tells the executor to poll instead of submit. I rebuilt both versions on a scratch PostgreSQL 17.11 database to be sure I was not misremembering, and the buggy claim does exactly what you would fear:

 status  | remote_request_id
---------+-------------------
 RUNNING | req-abc

The remote_request_id survives, the status does not. A recovered task that had already been paid for comes back looking like a fresh one. The guard at the top of runImageTask still catches it, because that guard checks the id as well, but at that point the query had quietly become the only thing standing between a restart and a double charge, and I would rather two things stood there.

On the same scratch database, SKIP LOCKED behaves as advertised: with three eligible rows and one of them held by a concurrent SELECT ... FOR UPDATE in another session, the claim returned 2 rows immediately instead of blocking on the third. That is the whole reason the clause exists, and it is worth confirming with your own eyes once, because the alternative failure mode, a supervisor tick that silently waits on a lock, looks identical to a supervisor tick that has nothing to do.

Concurrency is per kind, not global, because the kinds have nothing in common. Take four concurrent image slots, three for clips, two for speech and exactly one for the final local render: the remote ones are limited by the provider’s rate limits and the local one is limited by CPU. Those particular numbers are arbitrary and yours will differ. The structure is not arbitrary: one global limit for all four would either starve the cheap fast work or oversubscribe the machine.

Heartbeat, because not every step polls

A lease needs its own renewal, separate from whatever the work is doing. I learned this from a step that had no polling at all.

Remote generation renews the lease for free as a side effect: the poll loop runs every few seconds, and touching the row at the same time costs nothing. Local work has no such loop. The final render is one long call into ffmpeg that returns when it returns, and speech synthesis for a short passage is a single request that either answers or does not. Both of them held a lease that nobody was extending, so the supervisor watched the lease expire on a task that was working perfectly, decided the worker was dead, and started recovery on a row that was still being written to.

The fix is a timer independent of the work: renew at one third of the lease duration, so two consecutive misses still leave a margin before expiry. The ratio is what matters, not the absolute value. pg-boss reaches the same conclusion from the other direction and documents it as a separate mechanism from expiration, with a floor of 10 seconds on heartbeatSeconds and a monitor that only notices staleness on its own interval, 60 seconds by default. Detection time is the sum of the two, which is worth internalising before you tune the lease down to something clever.

Pinning the order with a test

The order of those three writes is load-bearing and invisible, so it gets asserted directly rather than implied by behaviour. The executor takes its ports as an interface, the test passes recording fakes, and the assertion is on the call log:

1
2
3
4
5
6
7

expect(calls).toEqual([
  "markSubmitStarted",
  "submitImage",
  "recordSubmitted",
  "poll",
  "complete",
]);

Then the recovery case, which is the one that actually costs money if it regresses:

1
2
3

// a task restored in SUBMITTED state with a saved request id
expect(calls).not.toContain("submitImage");
expect(calls).toEqual(["poll", "complete"]);

A test that asserts on a sequence of call names is usually a smell, because it pins implementation detail rather than behaviour. Here the sequence is the behaviour. Any refactor that moves markSubmitStarted after submitImage still passes every functional test, produces identical output on the happy path, and reintroduces the exact bug the design exists to prevent. The only way to catch it is to say out loud that the write comes first.

The price in the docs is not the price on the invoice

Store two numbers per task, estimated and actual, because the first one will be wrong. I built the cost display off a model catalogue populated from provider documentation, showed users an estimate before they pressed the button, and felt good about it. The first real run billed noticeably more than the catalogue said. Take a step documented at four cents and invoiced at six: that is half again as much, multiplied by every step in a job. An estimate that far off is worse than no estimate, because people plan around it.

The catalogue was not lying, it was quoting a headline number that did not include what the actual call resolved to. The correction is structural rather than a better catalogue: cost_estimated comes from the provider’s quote endpoint at submit time, cost_actual comes from the billing figure that arrives with the result, and the user-facing total is the sum of whichever is available per task. When the two disagree the actual value wins and the catalogue entry gets fixed. Documentation prices are for capacity planning. Billing figures are for arithmetic.

Bottom line

If a background task spends money at an external provider, the queue’s retry is no longer a reliability feature and the row, not the job, has to be the unit of recovery. Write the “I am about to spend money” boundary to the database before the request goes out, save the provider’s request id the moment it exists, and accept a third terminal state for the narrow window in between, where neither retrying nor failing is honest. The RFC has been describing this for years under “some means to detect that the original request was never applied”, and the means is a column.

Primary sources worth reading in full: RFC 9110 §9.2.2 on idempotent methods, the PostgreSQL SELECT reference on SKIP LOCKED, the pg-boss queue documentation on heartbeats versus expiration, Stripe on idempotent requests, and the retry predicates in openai-node and anthropic-sdk-typescript, which are worth twenty minutes of anyone’s time who is sending paid POSTs through them.