Home Glossary Latency

Latency - Page 45

Latency measures the delay between an AI request and the resulting response. Interactive generative systems often track time to first token, which reflects how quickly output begins, and time to last token, which captures total completion time. Latency comes from model computation, queueing, network transfer, retrieval, tool calls, safety checks, and application logic. It varies with prompt length, output length, model size, hardware, batching, cache use, and current load. Optimizing only average latency can hide poor experiences, so teams also monitor percentile values and separate each stage of the request. The right target depends on whether the product is conversational, analytical, offline, or safety-critical.

Nvidia Says $100B Investment Into OpenAI Is Likely Off the Table
By • 3 mins read
AI & Machine Learning, News, Startups & Investment

Nvidia Says $100B Investment Into OpenAI Is Likely Off the Table

By • 3 mins read

Nvidia CEO Jensen Huang said the company’s $30 billion investment in OpenAI could be its last before the AI startup pursues an initial public offering. The chipmaker also indicated its $10 billion investment in Anthropic may mark the end of its funding commitments to major AI model developers.