Skip to content
ai

Model serving

LLM Model Serving

Definition

Model serving refers to the infrastructure and software stack for deploying trained AI models to handle real-time inference requests. Key challenges include managing GPU memory for concurrent requests, batching for efficiency, and meeting latency SLOs.

Specialized serving frameworks like vLLM, TGI, and TensorRT-LLM use techniques such as continuous batching and KV cache management to maximize throughput.

Related Terms

vLLM
vLLM Inference Engine
TensorRT
TensorRT
Batch inference
Batch Inference
Inference
Model Inference
Quantization
Model Quantization
← Back to Glossary

Ship secure code faster

Crash Override integrates security into the developer workflow. No context switching, no waiting on reviews.

Talk to a Human See the Product