Latency
Latency is the time an AI system takes to begin or finish responding after it receives an input.
#LlmOptimization
Explore AIstify's latest reporting, research, and expert analysis tagged with "llm optimization", collected in one continuously updated archive.
Related Posts
KV Cache
A KV cache stores attention keys and values from earlier tokens so a transformer can generate later tokens more efficiently.