Batch inference

Batch inference processes multiple inputs simultaneously through a model rather than handling each request independently. Grouping requests into batches improves GPU utilization and throughput at the cost of increased latency for individual items.

Continuous batching, used by systems like vLLM, dynamically fills batches to maximize hardware efficiency for LLM serving.