Batch API
POST https://api-staging.impossiblecarrot.cc/api/v1/batches
GET https://api-staging.impossiblecarrot.cc/api/v1/batches
GET https://api-staging.impossiblecarrot.cc/api/v1/batches/{id}
POST https://api-staging.impossiblecarrot.cc/api/v1/batches/{id}/cancelRequires a bearer API key. A batch is a set of requests to one endpoint, submitted together and run in the background, for work that does not need an answer within the request that asked for it.
A batch changes how requests are submitted and nothing about how they are served. Every request in it is routed, held against your balance, settled and recorded as the same body posted to the endpoint would be, under the key that submitted the batch and its spend limit, and it carries a generation of its own. A batch earns no discount: it costs what its requests cost one by one.
Submit a batch
endpoint names the request shape every item is written in: /v1/chat/completions,
/v1/responses or /v1/messages. Each item is a custom_id of your own, unique within the batch,
and the body that endpoint takes. The route answers 202 once the batch is stored, with
status: "validating"; nothing has run or been charged yet.
curl ${API_BASE_URL}/batches \
-H "Authorization: Bearer $HEIMDALL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"endpoint": "/v1/chat/completions",
"model": "acme/alpha-v1",
"completion_window": "24h",
"requests": [
{ "custom_id": "req-0001", "body": { "messages": [{ "role": "user", "content": "Summarize the first report." }] } },
{ "custom_id": "req-0002", "body": { "messages": [{ "role": "user", "content": "Summarize the second report." }] } }
]
}'
{
"id": "batch_5d1f0c9e2a7b4e36b8a4c1d27e90f3a5",
"object": "batch",
"endpoint": "/v1/chat/completions",
"model": "acme/alpha-v1",
"completion_window": "24h",
"status": "validating",
"created_at": 1786881571,
"expires_at": 1786967971,
"finalized_at": null,
"request_counts": { "total": 2, "completed": 0, "failed": 0 },
"results_file_id": null,
"results": null,
"error": null
}
| Member | Notes |
|---|---|
endpoint | Required. One of /v1/chat/completions, /v1/responses, /v1/messages. |
requests | Required. 1 through 1000 items of { custom_id, body }. A custom_id is up to 128 printable characters and appears once in the batch. |
model | Optional. A model for every request. A body that names none takes it; a body that names a different one fails on its own. |
provider | Optional. only and allow_fallbacks, applied to every request whose body states no provider of its own. |
completion_window | Optional. 24h, which is also the default. |
A submission may be up to 32 MiB, which is more than an ordinary request body. stream is not
accepted in a body, because every request is answered in full, and a chat body cannot ask for
store: true.
Bodies are checked one at a time
The envelope is checked as a whole: a repeated custom_id, an endpoint the API does not batch or a
missing requests refuses the submission with 400. A body is checked against the chosen
endpoint's schema as the batch is accepted, and one that does not parse fails on its own. It is
reported in the results with the error the live route would have answered, and counted in
request_counts.failed; the rest of the batch runs. A batch in which every body fails ends failed
with the reason in error.message.
How the requests run
Batches of an account run oldest first, and at most eight of an account's requests are in flight at once, across every gateway. Requests are sent in submission order, and what each does is what the same body posted to its endpoint does:
- It is routed over the live catalogue and the workspace's routing defaults as they stand when the request is sent, and held against the balance of the key's account.
- A request the account cannot fund, or that no endpoint serves, fails on its own with that answer, and charges nothing.
- A key that was disabled, expired or moved out of its workspace after it submitted the batch
authorizes nothing: its remaining requests fail with
401. - The spend limit of the submitting key applies to each request as it is sent.
- The
X-TitleandHTTP-Refererheaders of the submission name the application every request is attributed to.
A gateway that stops while a batch runs does not lose it. Another gateway takes the account over once
the first one's claim runs out and settles whatever the first left. A request that was never charged
is sent again; a request that was charged and whose reply the first gateway never recorded is
not sent again, since that would charge twice. It fails with type: "server" and names the charge in
error.request_id.
Statuses
| Status | What it means |
|---|---|
validating | Stored and waiting for a gateway to take it. |
in_progress | Being sent. |
cancelling | A cancellation was asked for; the requests in flight are being settled. |
completed | Every request has an outcome, an error included. |
failed | No request was runnable: every body failed validation. |
expired | The completion window closed first. Requests not yet sent were not sent. |
cancelled | The batch was cancelled. Requests that had not finished were not charged for. |
The last four are terminal. When the window closes, a request already in flight is allowed to
finish, and the ones not yet sent are listed in the results as batch_expired.
Read a batch
GET /batches/{id} returns the batch with its counts. request_counts.completed and .failed are
the requests that ended in each way; a request stopped or expired before it was sent is in neither, so
total less the two is the number the batch did not run.
curl https://api-staging.impossiblecarrot.cc/api/v1/batches/batch_5d1f0c9e2a7b4e36b8a4c1d27e90f3a5 \
-H "Authorization: Bearer $HEIMDALL_API_KEY"GET /batches lists the account's batches, newest first. status may be repeated to include
several, created_after and created_before take unix seconds, a date or an ISO 8601 instant and
exclude it, and limit is 1 through 100 and defaults to 20. To read the next page, pass last_id as
after; there is no offset. A list carries results: null on every batch.
curl "https://api-staging.impossiblecarrot.cc/api/v1/batches?status=completed&status=failed&limit=20" \
-H "Authorization: Bearer $HEIMDALL_API_KEY"Results
When a batch ends it writes its results to a file in your file store, named by
results_file_id, with one JSON line per request in submission order, and clears the requests and
replies it held. Read the file from GET /files/{id}/content. A retrieved batch carries the same
lines inline as results for as long as the file exists.
{"id":"breq_7e0c1a94d3b84f2a9c56e81d0b3f2a47","custom_id":"req-0001","response":{"status_code":200,"request_id":"gen-1786881609-aBcDeFgHiJkLmNoPqRsT","body":{"id":"gen-1786881609-aBcDeFgHiJkLmNoPqRsT","object":"chat.completion"}},"error":null}
{"id":"breq_a3c4e5f6071829304a5b6c7d8e9f0a1b","custom_id":"req-0002","response":null,"error":{"type":"invalid_request","message":"Invalid request: messages: Too small: expected array to have >=1 items","param":null,"code":400,"request_id":null}}
Exactly one of response and error is set on a line.
response.bodyis what the endpoint answered, in the shape its route answers in.response.request_idis the id the request was billed under, the valueX-Generation-Idcarries on a live request, so a line can be matched to its generation and its ledger entry.error.typeanderror.codeare those of the answer the live route would have given.batch_cancelledandbatch_expiredmark a request the batch ended without sending, with acodeofnull.- A request whose reply would not fit is failed rather than cut short. A single reply may be up to 1 MiB
and all of a batch's replies together fit one stored file; a request the file has no room left for is
failed with
results_fulland is not sent, so it is never charged for a reply with nowhere to go.
The results file is a stored file like any other: it expires 48 hours after the batch ends, and
from then on results is null, the file answers 404 and the batch still reports how it ended and
its counts. Keep your own copy of anything you need for longer.
Cancel a batch
curl -X POST https://api-staging.impossiblecarrot.cc/api/v1/batches/batch_5d1f0c9e2a7b4e36b8a4c1d27e90f3a5/cancel \
-H "Authorization: Bearer $HEIMDALL_API_KEY"Dispatch stops at once and the requests in flight are cut. The batch is cancelling until what ran
has settled, then cancelled. A request that completed keeps its reply and its charge; one that was
cut or had not been sent is listed as batch_cancelled and is not charged, unless it was charged before
it was cut, in which case error.request_id names the charge. Asking again while the
batch is stopping, or after it has stopped, returns it unchanged; a batch that ended another way
answers 409.
Refusals
| Status | When |
|---|---|
400 | The envelope did not parse, a custom_id repeats, or a list parameter is outside its range. |
401 | The key is missing, unknown, revoked or expired. |
404 | No such batch on your account. An unknown id, a malformed one and another account's give the same answer. |
409 | The batch already ended as completed, failed or expired, so it cannot be cancelled. |
413 | The submission is larger than 32 MiB. |
503 | The keys the batch's requests are stored encrypted under could not be reached. Nothing was stored; repeat the submission shortly. |
429 | The account already has 10 batches waiting or running, or the key is over its request rate. metadata carries limit and active for the first. |
Your batches are your data
The requests a batch holds, and the replies it collects, are kept only while the batch needs them. They
are cleared in the same step that writes the results file, and the file expires 48 hours later. What
remains of a batch is a record of what ran: its status, its counts, the custom_id of each request and
the id it was billed under. The account export carries every batch and its
requests, with the request bodies for as long as the batch runs, and every batch is erased with the
account when a deletion runs.
While the batch runs, each request and each reply is stored encrypted, under a key that is kept outside
the database and destroyed no more than 50 hours after the value was written, so a backup of the database
taken in that time cannot give them back afterwards. If the key for a request is lost before its turn, the
request is not sent and is not charged: its result is a server error saying the stored request could not
be read. If the key for a reply is lost after the request was charged and answered, its result is a server
error whose request_id names the charge, and the request is not run again.