Questions & explanations
1. Why is navigating a robot inside a volcano different from navigating in an office building?
A volcano has extreme heat, corrosive gas, and rough terrain. Regular robot sensors like cameras may be destroyed or blinded by smoke. Office sensors are designed for normal temperatures and clear air. In a volcano, robots need specialized sensors that can withstand high heat, like thermal cameras and laser rangefinders that work through smoke. The robot must also have robust algorithms that handle slippery ash and steep slopes. Communication may be poor, so remote operation is difficult. The robot may need to operate semi-autonomously, making its own decisions because commands from humans are delayed. All these differences require completely different hardware and software.
2. What makes continuous control tasks hard for deep reinforcement learning?
Continuous control tasks have action spaces with many real-number values, like moving a robot arm with multiple joints. Deep RL must choose actions from infinite possibilities, unlike discrete games with few moves. This requires algorithms that can output continuous actions, such as DDPG or SAC. Exploration also becomes harder because the agent must try many different action combinations to learn. The high-dimensional state sensors, like cameras or joint angles, add complexity. Deep neural networks help approximate optimal policies but need careful tuning to avoid instability. Sample efficiency is a big challenge because continuous tasks often demand millions of steps.
3. A deep-sea robot needs to navigate in total darkness. What sensing challenges does it face?
In deep sea, sunlight does not reach, so standard cameras are useless. The robot must use sonar (sound waves) to map its surroundings. Sonar sends out sound pulses and listens for echoes to detect objects. However, sound travels slower than light and can be distorted by water temperature and pressure. The robot also faces high water pressure that can damage electronics, so it must be housed in a strong pressure vessel. Navigation is often done with inertial sensors and acoustic beacons, because GPS does not work underwater. The robot must also deal with currents that push it off course. Thus, navigation relies on careful sensor fusion and robust control.
4. A robot has a sensor that gives noisy measurements of a moving car. How does the robot handle this noise when predicting collision?
The robot does not treat the sensor reading as exact. Instead, it uses a technique like a Kalman filter to combine the noisy measurement with a prediction of the car's motion. This produces an estimate of the car's position with an uncertainty—a range of where it might really be. Over time, the uncertainty grows because small errors in speed and direction accumulate. When computing collision probability, the robot considers this entire uncertain range. If there is even a small overlap between its own uncertain position and the car's, the probability may be non-zero. The robot then decides if that probability is low enough to continue.
5. What is the distribution shift problem in behavioral cloning, and how can it be fixed?
The distribution shift problem happens when the robot's own actions lead it to states not present in the expert data. For example, if the car drifts slightly left, it sees a new road view that was never in training. Then the learned policy makes poor decisions and the error grows. One fix is to collect more demonstrations that cover recovery actions, but that's expensive. Another approach is to let the robot interact with an environment and correct its mistakes using online training. Data augmentation also helps by artificially creating off-trajectory examples. DAgger is an algorithm that interleaves expert feedback to correct drifts.
6. How do space robots like Mars rovers handle the communication delay when being controlled from Earth?
The delay between Earth and Mars can be several minutes. This means remote control is impossible—the robot cannot wait for commands every second. Instead, the rover is given a high-level plan for the day, like 'drive to that rock and take a picture'. It then uses its own sensors and onboard computers to navigate autonomously, avoiding rocks and slopes. It executes the plan step by step, checking safety at each move. Only after completing the plan or encountering a problem does it send data back to Earth. The next day, humans send new commands based on that data. This blend of autonomy and human oversight is key for space exploration.
7. What does it mean for a navigation algorithm to be 'provably safe'?
A provably safe algorithm comes with a mathematical proof that it will never cause a collision or enter forbidden areas. Proofs use methods like reachability analysis, which computes all possible positions the robot could reach. If none of those positions are dangerous, the motion is declared safe. Another method is barrier certificates—functions that decrease when the robot approaches danger. The proof guarantees safety for all possible scenarios within the model. However, the proof is only as good as the model; unexpected conditions can break it. So engineers must carefully model the environment and the robot's dynamics.
8. A robot must navigate through a crowd of people. Why might a formal safety guarantee be hard to achieve?
Formal safety requires a precise mathematical model of everything that could happen. People are unpredictable—they can change direction, speed, or stop suddenly. It is very hard to model all possible human behaviors accurately. If the model misses some behaviors, the safety proof might not hold in reality. For example, a child might run into the robot's path unexpectedly. Engineers often use conservative assumptions, like assuming people move at maximum speed. But even then, the proof may only cover a limited set of scenarios. So formal guarantees are easier in controlled environments than in dynamic, human-filled spaces.
9. A self-driving car must decide whether to change lanes. How can it use risk assessment to make this decision?
The car first predicts the future motion of all nearby vehicles using uncertainty propagation. It calculates the probability of collision if it stays in its lane and if it changes lanes. It also considers the severity—a crash at high speed is more dangerous. Risk is often defined as probability times severity. The car then chooses the action with the lowest risk. For example, staying in lane might have a 1% collision chance with a slow car, while changing lanes might have a 5% chance with a fast truck. Even though 5% is low, the severity is higher, so the car may stay. This trade-off is done continuously.
10. Explain maximum entropy inverse reinforcement learning (MaxEnt IRL) in simple terms.
MaxEnt IRL assumes that the expert's demonstrations are not perfect but have some randomness. It looks for a reward function where the expert's trajectories have the highest probability among all possible behaviors. The 'maximum entropy' part means it chooses the reward that makes the expert's actions as likely as possible while still allowing other actions to be possible. It uses a softmax model: better reward trajectories are exponentially more likely. The algorithm iteratively adjusts the reward to match the expert's state visitation frequencies. This avoids overfitting to noise in the demonstrations.
11. Define behavioral cloning and give an example of its use in robotics.
Behavioral cloning is a kind of imitation learning where a robot watches expert demonstrations and directly copies the expert's actions. The robot records the expert's decisions (e.g., steering angles) along with the observed states (e.g., camera images). It trains a neural network to predict the expert's action from each state. For example, a self-driving car learns to follow lane markings by watching a human driver's steering wheel angles. This is simple but can fail if the robot encounters new situations not seen in training. It is like learning by mimicking without understanding the reasons.
12. Why is sample efficiency important in deep RL for continuous control, and how can it be improved?
Sample efficiency means getting good performance with fewer environment interactions. In continuous control, each trial is slow or expensive, like a real robot moving. Poor sample efficiency requires millions of steps, which is impractical. Methods to improve include using off-policy algorithms like SAC that reuse old data efficiently. Also, model-based RL learns a world model to simulate outcomes and plan. Hindsight experience replay helps by relabeling failed goals as achieved. Another trick is to use data augmentation or pretraining. Still, sample efficiency remains a major research focus.