#Latency

A KV cache stores attention keys and values from earlier tokens so a transformer can generate later tokens more efficiently.

Inference is the process in which a trained AI model applies learned patterns to new input and produces a prediction, decision, or generated response.