--- title: "Batches and Limits" description: "Submit up to 1000 sources in one batch, keep their outputs apart, deduplicate retries with idempotency keys, and work within the transcoding rate and volume limits." category: "Deployment" language: en source: https://cubepath.com/docs/products/deployment/transcoding-batches updated: 2026-11-17 --- # Batches and Limits One video at a time is fine while you're wiring things up. A back catalogue isn't, so the **New Job** form has a **Bulk** mode, and the API has a batch endpoint behind it: one submission, up to **1000 sources**, all sharing the same destination and the same output spec. ## Bulk mode Switch the mode to **Bulk** and the single-source fields turn into a list. One source per line: - **URLs**: one `http(s)://` URL per line. - **S3 object keys**: one key per line, all read from a single **shared source bucket** whose endpoint and credentials you enter once. Everything else (the destination bucket, the prefix, the outputs, the webhook) is defined once and applied to every job in the batch. Each line becomes its own independent job: they queue separately, run separately, and one failing doesn't touch the others. ## How outputs are kept apart In **Bulk** mode, every job in a batch writes into **its own subfolder** under the destination prefix, derived from the source's filename and its position in the list. A batch with `intro.mp4` and `chapter-1.mov` under the prefix `course/` produces something like: ``` course/1-intro/output_0.mp4 course/2-chapter-1/output_0.mp4 ``` The name is sanitized (anything that isn't a letter, digit, dot, hyphen or underscore becomes `_`, and long names are truncated), so odd characters in your filenames can't produce awkward keys. The leading number keeps two sources with the same basename from colliding. > Note: This is the part people get wrong when they build batches by hand through the API, because the API doesn't add the subfolder for you. Without a distinct sub-path per input, every job writes `output_0.mp4` to the same key and you end up with one surviving file out of a thousand. ## Idempotency A job submitted on its own (**Single** mode, not a batch) accepts an **idempotency key**. Send the same key twice and the second request returns the existing job instead of creating a duplicate. Use it whenever the caller might retry: a queue worker that redelivers on timeout, a webhook handler that can fire twice, a form a user can double-submit. A stable key derived from the source (the object key, or your own asset ID) makes retries free and turns "did that submit?" into a question you don't have to answer. ## Limits | Limit | Value | |-------|-------| | Outputs per job | 20 | | Inputs per batch | 1000 | | Active (unfinished) jobs per organization | 20,000 | | Create-job rate | 60 / minute | | Batch rate | 10 / minute | The active-jobs cap counts everything that hasn't reached a final state. It's generous, but it's the one you'll hit if you fire twenty full batches back to back: 20,000 jobs queue up faster than they encode. ## Working through a large catalogue - **Submit in batches, not in a loop.** One batch of 1000 costs one request against the batch limit; 1000 individual creates cost you 17 minutes of waiting on the per-minute rate. - **Pace yourself against the active cap.** Submit the next batch when the queue has drained, not when the previous request returned. - **Set a webhook.** At this volume, polling the list is the wrong approach: let the jobs tell you they're done. See [Job status](/docs/products/deployment/transcoding-job-status). - **Keep the output spec small.** Every extra rendition multiplies across every input. A three-rung HLS ladder on 1000 files is 3000 encodes. > Tip: Test the exact output spec on one file first. The most expensive mistake in this product is discovering after a 1000-file batch that the ladder was missing a rung, or the prefix was wrong. See [Create a job](/docs/products/deployment/transcoding-jobs) and [Outputs](/docs/products/deployment/transcoding-outputs). ## If a batch is rejected | Symptom | Likely cause | |---------|--------------| | "Add at least one source line" | The list is empty, or every line was blank | | Destination bucket required | The shared destination is missing its bucket | | Rate limited | You hit the batch or create rate (wait a minute), or the active-jobs cap (wait for jobs to finish) | | Rejected only when creating jobs | The organization is unverified, suspended, or out of balance |