What Is Batch Processing? Definition, Examples, and When to Use It

Batch processing collects data over a period of time and then processes all of it at once as a single job, usually on a schedule, instead of handling each record the moment it arrives. A payroll run that reads a month of timesheets on the last day of the month and produces every paycheck in one pass is a batch job. The defining features are a bounded input, a scheduled start, and a design that favors total throughput over the speed of any single record.

What Batch Processing Means

A batch is a set of records that is complete before the work starts. The job reads the whole set, applies the same steps to every record, and writes the results out at the end. Nothing in the batch counts as done until the job finishes, so a batch job is measured by how long the whole run takes rather than by how quickly one record is handled.

The term "bulk processing" is often used for the same idea, and "batch computing" or "batch mode" describes a system that runs this way. Early mainframes worked almost entirely in batch mode, and the pattern is still the standard way to run reports, billing, and data loads today.

Examples of Batch Processing

• Nightly payroll runs. Hours are collected all month and every employee is paid in one job.
• End-of-day bank settlement. A bank nets the day's transfers between accounts after the business day closes.
• Monthly billing. A utility or a software company reads a month of usage and produces every invoice at once.
• ETL loads into a data warehouse. ETL stands for extract, transform, load: the job pulls records from source databases, reshapes them, and writes them into the warehouse on a schedule.
• Report generation. Daily sales reports and weekly summaries are built from the previous period's data.
• Model training. A recommendation model is retrained each night on the day's clicks and purchases.
• Batch image processing. A photo service resizes and watermarks every upload from the day in one job rather than one image at a time.

Characteristics of a Batch Job

The input is bounded. The job knows the full set of records before it starts, so it can split the work across machines and report exact progress.

The start is scheduled. A batch job runs at a fixed time, such as 2 a.m. every night, or when a trigger fires, such as the moment the day's input files are available.

Throughput matters more than latency. Throughput is the amount of work finished per unit of time, and latency is the delay before one record is handled. A batch job accepts hours of latency in exchange for processing millions of records per hour at low cost.

Retries are simple. If the job fails halfway, the usual fix is to rerun it, because the input has not changed. Batch jobs are therefore written to be idempotent, which means running them twice produces the same result as running them once.

Tools for Batch Processing

Small jobs need nothing more than cron, the scheduler built into Unix-like systems, and a shell or Python script. Large jobs that must be split across many machines use Apache Spark or Hadoop MapReduce, both of which divide the input into partitions and process the partitions in parallel. When a pipeline has many dependent steps, an orchestrator such as Apache Airflow runs them in the right order and retries the ones that fail. Cloud providers offer managed versions of the same idea, such as AWS Batch and Google Dataflow running in batch mode.

Batch Processing vs Stream Processing

Stream processing is the opposite pattern: each record is handled within milliseconds of the time it is produced, and the input never ends. The full comparison, including how to choose between them, is in What is batch processing vs. stream processing and when should you use each?. This page covers only the batch side.

The diagram below shows the two patterns side by side.