Follow a robot from camera input to movement and feedback, and learn how training, simulation and real-world tests separate a demo from reliable autonomy.
Ask a robot to put a mug on a tray and the instruction sounds almost trivial. The machine still has to find the right mug, work out where it can grasp it, avoid nearby objects, lift without slipping and check that the mug actually reached its destination. A fluent explanation of the task does none of those things by itself.
Physical AI is artificial intelligence that helps machines perceive, make decisions and act in the physical world. In robotics, it connects learned capabilities with sensors, control software and mechanical hardware. This guide follows an illustrative mug-moving task from the first camera image to the final check, then explains how such systems are trained and what separates an impressive demonstration from dependable autonomy.
What Physical AI Means in Robotics
Nvidia’s definition covers systems that sense, reason and act in their surroundings, including robots and autonomous vehicles. The term describes a broad approach, rather than a particular model architecture or a requirement that a machine look human. A wheeled warehouse robot and a stationary picking arm can both use physical AI.
Robotics is the wider engineering discipline: it includes the motors, mechanisms, electronics and software that make a machine operate. Physical AI contributes learned abilities within that system. Conventional automation can remain the better choice when a task is predictable and the environment can be tightly controlled; adding a language model does not automatically make a robot more useful.
For our example, imagine a robot arm beside a table, with a camera looking at a blue mug and an empty tray. The instruction is “put the blue mug on the tray.” The following stages are a way to understand the job; actual systems may combine several of them inside one model rather than expose five separate software modules.
Stage
Question to answer
Mug task
Sensing
What information is available?
Capture the table and arm position.
Understand
What matters for the goal?
Identify the blue mug and tray.
Planning
What should happen next?
Choose a grasp and clear route.
Movement
How should the body move?
Reach, grip, lift and place.
Feedback
Did the action work?
Check grip and final placement.
First, the Robot Needs More Than a Picture
In the example, the camera supplies a view of the tabletop. That view contains useful clues about color, edges and relative position, but it leaves questions unanswered. Is the mug behind the tray or beside it? Is the handle hidden? Could a reflection be mistaken for an opening?
A robot designer might add a depth camera, a second viewpoint or a camera near the gripper to reduce those ambiguities. The arm also needs information about its own configuration, such as joint positions. An image showing a mug near the center of the frame does not, on its own, specify where the arm should move.
Consider what happens if someone shifts the camera after the system is set up. The same image location may now correspond to a different point on the table. For this illustrative task, consistent sensor positioning and calibration are part of making observations useful for action. Better object recognition cannot compensate for every mismatch between the robot’s view and its body.
Understanding Connects Words With Objects
The instruction identifies an object and a destination. The robot must connect “blue mug” with something in its observations and “tray” with a place where the mug can be set down. If there are two blue mugs, a useful system should resolve the ambiguity or ask a question, rather than silently choosing whichever object its model happens to prefer.
Vision-language-action models, or VLAs, bring visual observations, language instructions and robot actions into one learning framework. DeepMind’s earlier RT-2 research combined web and robotics data to predict actions from camera images and instructions. The important distinction is that the output can represent a robot action, rather than only a sentence about what a robot ought to do.
That does not mean every VLA builds an explicit, complete 3D map or understands an object as a person does. For the mug example, recognizing the correct object, estimating a usable grasp and noticing uncertainty are separate practical requirements. A model can succeed at one and fail at another.
Planning Turns the Goal Into Feasible Motion
A possible task plan is to approach the mug, close the gripper, lift, move above the tray, lower and release. A motion planner tackles the geometry: can the arm follow a route without hitting the table, tray or its own structure? Northwestern University’s motion-planning overview distinguishes a collision-free path from a trajectory that also specifies timing.
Imagine a tall bottle standing between the mug and tray. Moving straight toward the destination could strike it even if the starting and finishing positions are valid. Our robot might need to lift higher or travel around the bottle; if neither route fits its reach and motion limits, the proposed task needs to change.
Some systems separate a high-level planner from a learned movement policy. A policy is the rule or model that chooses actions from observations, sometimes also conditioned on a goal. Other systems learn more of the sequence together. These architectural choices change where decisions happen, but the physical constraints remain.
Movement Still Depends on Controllers and Feedback
An action might specify target joint positions or a desired gripper position and orientation. Hugging Face’s LeRobot documentation explains both representations. Moving the gripper to a point in space may require inverse kinematics, which calculates suitable joint configurations for that target. A robot with different arm lengths or joints cannot simply inherit identical motor commands.
The controller then uses sensor measurements to adjust actuator commands. This is closed-loop control: measured behavior is fed back into the system instead of assuming the requested movement happened perfectly. The Modern Robotics control overview describes these loops and the different objectives of controlling movement and controlling force. Low-level control can run much faster than a model making high-level task decisions.
In our example, closing the gripper is only an attempted grasp. The robot might check whether the mug moved with the hand, whether contact was detected or whether the gripper closed unexpectedly far. Depending on its sensors and software, a failed grasp could lead to another observation, a different attempt or a request for assistance.
The final check matters just as much. Releasing the mug above the tray does not prove it landed upright or remained there. Feedback should connect the instruction to an observable result. A system that announces success because it finished its planned sequence can be confidently wrong.
Where Gemini Robotics 2 Fits
Google DeepMind’s July 2026 robotics announcement illustrates this layered approach. Gemini Robotics 2 is a VLA for motor control, Gemini Robotics ER 2 handles embodied reasoning and task orchestration, and Gemini Robotics On-Device 2 is optimized for local operation. The work includes humanoid whole-body movement and manipulation with different hands and grippers.
The company’s published results also show uneven capability. For Apollo with Inspire hands, it reports a 45.7% average success rate for the “pick up from floor” skill category. That is a result for a particular evaluation, not a universal failure rate for every task or robot. It nevertheless helps explain why a successful clip cannot establish general reliability.
How Robots Learn From Training Data
One route is imitation learning: train a policy on examples of a task being performed. An operator can teleoperate a robot while its observations and actions are recorded. Hugging Face’s training tutorial takes this process through recording demonstrations, training a policy and evaluating it on hardware.
For the mug task, a demonstration needs to connect what the robot observed with what it did. LeRobot’s dataset format organizes multimodal time-series data, sensorimotor signals and camera video. This makes robot learning data different from a folder of unrelated photographs: the relationship between observation, action and time is central.
As an illustrative dataset-design problem, imagine recording every demonstration with the mug in exactly the same spot. A policy might learn a narrow routine that works beautifully there. Moving the mug, changing its handle orientation or introducing clutter would then test something the recordings did not adequately cover. More recordings of the identical scene would not necessarily solve that problem.
Training data should reflect the intended operating conditions, while evaluation should include situations withheld from training. In the mug example, those might include unfamiliar mugs and different arrangements of nearby objects. The useful question is whether the learned behavior transfers, rather than whether the model can reproduce a familiar sequence.
What Simulation Can Teach—and What It Cannot
Simulation lets developers create virtual robots and environments, generate observations and test actions without risking physical equipment on every attempt. Nvidia’s Isaac Sim supports physics simulation and synthetic data generation, including variation in lighting, reflections and object placement. A virtual version of our task could expose the robot to many table arrangements.
Reinforcement learning provides another training route: a policy is optimized using a reward signal for its behavior. In the mug example, a poorly chosen reward might favor moving the mug quickly while overlooking unstable placement. That hypothetical failure shows why defining success requires more care than rewarding visible progress toward a destination.
The simulation-to-real gap is the mismatch between virtual conditions and hardware. A simulated grasp might behave differently when the real mug is slippery, the camera is noisy or the mechanism has unmodeled friction. Research on domain randomization varies simulated appearances so that models are less dependent on one artificial setting. Such variation can help transfer, but does not prove it will work in every real environment.
Generative world models add a related possibility: predicting how an observed scene could evolve. Our coverage of Odyssey-3 explores that direction. For evaluation, however, a convincing generated scene and a measured hardware outcome are different evidence. A video that looks physically plausible cannot establish that a real gripper will hold a particular mug.
Why a Demonstration Is Not Reliable Autonomy
A demonstration establishes that a system completed a task under the conditions shown. Without additional information, it does not establish how many attempts were made, whether an operator intervened, how much preparation occurred or how the system behaves when something changes. Those omissions can matter more than how smooth the successful run looks.
Long tasks make the distinction especially clear. In a simplified illustration, suppose each of 20 steps succeeds independently with a probability of 95%. The probability of all 20 succeeding is about 36%, calculated as 0.95 raised to the twentieth power. This is not a measured result for any robot; it shows how strong individual-step performance can coexist with weak end-to-end completion.
Real failures are not necessarily independent, and a robot may recover from them. That makes recovery behavior part of the evaluation, rather than a footnote. A missed grasp followed by a safe retry is different from a missed grasp followed by a false success message. Both are also different from an operator quietly rescuing the task.
For the mug example, meaningful real-world testing would repeat the task across the intended range of objects and conditions, count unsuccessful attempts and record interventions. It would examine completion time and damage as well as success. A useful deployment boundary might permit only empty mugs on a clear table, rather than claim unrestricted kitchen autonomy.
Questions to Ask Before Trusting a Physical AI System
When assessing a robot announcement or a proposed deployment, look for evidence that connects the advertised capability with the work the machine is expected to perform. These questions help make that comparison concrete:
What is the operating boundary? Ask which objects, environments and tasks are supported, and which remain outside scope.
How were results counted? Look for the number of trials, the definition of success and whether failures or restarts were excluded.
What did humans do? Distinguish autonomous execution from teleoperation, assistance and scene preparation.
Were conditions unfamiliar? Check whether evaluation used new objects, layouts or situations rather than only training examples.
What happens when it is wrong? Examine stopping, recovery, requests for help and how the system detects an unsuccessful action.
What does deployment require? Consider supervision, maintenance and the cost of interventions alongside task speed.
Safety also needs its own evidence. A task-completion benchmark is not a safety certification, and an AI model’s ability to recognize danger does not establish that the entire machine can avoid it. Risk assessment must concern the deployed robot, its surroundings and the people who may encounter it.
For physical AI, progress becomes useful when better perception and decision-making translate into repeatable outcomes. The mug belongs on the tray, the robot needs to know whether it got there, and the people relying on the system need an honest account of when it works. That standard is less spectacular than a highlight reel, but far more valuable.
