Daily AI Paper

Reinforcement learning has made impressive progress on high-dimensional robots, including legged systems and humanoids that can learn agile behaviors. But once you want to deploy those behaviors around people, equipment, or other protected regions, safety becomes much harder. It is no longer enough to ask whether the current robot state is safe right now. You also have to ask whether there exists a future sequence of actions that can keep the robot safe, because in contact-rich, underactuated systems, a state that looks acceptable at one instant may still be doomed if the robot cannot steer away in time. And if a disturbance makes violation unavoidable, then the right goal is not simply “be safe,” but rather “minimize the harm while still doing the task as well as possible.” That is the core motivation behind this paper. The authors focus on feasibility-aware safe reinforcement learning, which is a framework that separates states into two regimes. In feasible states, there exists some policy that can avoid unsafe regions indefinitely, so the controller should maximize task return while staying safe. In infeasible states, safety can no longer be guaranteed, so the controller should instead reduce the severity of the violation as much as possible. This distinction matters because many standard safe RL methods do not really respect it. They combine task reward and safety into the same policy objective using penalties, Lagrange multipliers, or alternating updates. That sounds reasonable, but it creates a real optimization conflict: the action that improves task reward may be different from the action that reduces harm, so the two signals fight each other inside the policy update. Penalty methods can be too weak or too conservative depending on the weight. Primal-dual methods remove manual tuning, but replace it with a saddle-point optimization that can be unstable. So even when the method knows whether a state is feasible, safety still competes with performance during learning. A different family of methods uses safety filters. Instead of changing the policy objective, a safety filter takes the action proposed by a task policy and modifies it before execution. This is appealing because it separates concerns: the task policy focuses on reward, and the filter enforces safety at action time. Classical versions often rely on an analytic safety function and a known dynamics model, typically through control barrier functions and a quadratic program. That works well when the model is available and the action space is manageable, but it becomes difficult for high-dimensional systems. Some newer methods learn the safety function, which removes the need for an explicit model, but they still often require online optimization or action sampling at every step. That can be expensive as the action dimension grows. Another issue is that a filter that only minimizes instantaneous deviation from the nominal action is short-sighted: it may preserve the current action too aggressively even if a different action would improve long-term return after filtering. And there is a final edge case that classical projection-based filters do not handle well: if no safe action exists, then “project onto the safe set” is undefined, and the method does not tell you what the least harmful action should be. FAITH, which stands for Feasibility-Aware safety-fIlTered RL for High-dimensional systems, is designed to address all of those issues together. The central idea is to combine a learned safety critic, a feedforward safety filter, and a task policy trained through that filter. Instead of putting safety directly into the task-policy objective, FAITH changes the dynamics the policy experiences by filtering its actions. So the task policy is optimized only for reward, but under the filtered system. That means safety is enforced at execution time rather than as a competing term in the policy update. The method begins by learning an optimal safety critic, which is an action-value function for safety rather than reward. A state-value function can tell you whether a state is feasible, but it cannot evaluate candidate actions directly. The safety critic fills that gap by estimating whether taking a particular action from a state can still lead to a persistently safe future. The paper uses a backup actor to help train this critic. Intuitively, the backup actor supplies low-harm actions for the critic’s Bellman target, so the critic can learn the discounted safety fixed point from replay data. Because this is a learned approximation, the authors are careful to note that it is a practical safety estimate rather than a formal certificate. They also mention techniques like ensembles and target smoothing to improve the critic’s accuracy. Once the critic is learned, it defines an admissible-action condition: actions are considered acceptable if the learned safety value plus a small margin is below zero. But directly solving the corresponding nonlinear projection online would be costly, especially in high-dimensional action spaces. So FAITH amortizes that projection with a feedforward residual network. In other words, instead of running an expensive optimizer at every control step, the filter is trained to output an action that is close to the nominal one when the nominal action is already safe, and otherwise moves toward a safer action. The filter is trained with a penalty objective that balances two terms: staying close to the nominal action and satisfying the learned safety condition. The training schedule starts near the identity map and gradually increases the weight on safety, so the filter becomes more conservative over time. This design also gives FAITH a graceful behavior in infeasible states. If the learned admissible set is empty, then the filter cannot preserve safety because no action satisfies the learned condition. Instead of failing or switching to a separate recovery controller, the same objective naturally becomes a tradeoff between action deviation and predicted safety value. As the weight on the safety term grows, the filter approaches the action that minimizes predicted peak harm. The paper states this formally under standard assumptions such as compact action spaces and continuity, and the bound shows that the filter approaches the minimum of the learned safety critic at a rate on the order of one over the penalty weight. The important practical point is that the same controller remains well-defined whether the state is feasible or infeasible. The third component is the task policy itself. During training, the policy proposes nominal actions, the filter modifies them, and the environment responds to the executed actions. The policy is then optimized with PPO on the filtered rollouts, using only the task reward. Safety does not appear as a separate penalty in the reward function. That is one of the key conceptual differences from objective-based safe RL: the policy learns to succeed in the world as actually filtered, rather than trying to jointly optimize reward and safety in the same update. The paper states an equivalence result for the idealized case of a perfect filter. If the filter perfectly enforces safety on feasible states and leaves already admissible actions unchanged, then optimizing reward through the filtered dynamics recovers the constrained optimum on feasible states. That gives a clean theoretical interpretation of the method: with a perfect filter and global optimization, FAITH solves the right constrained problem without putting a competing safety term into the task-policy objective. The experiments are designed to test both the feasible and infeasible regimes, and to see whether the learned filter can scale to high-dimensional control. The authors evaluate on a double integrator example, a Safety Gym environment, and a 29-DoF humanoid. They also demonstrate the learned policies on a real Unitree G1 humanoid. The metrics are chosen to reflect the two regimes: in feasible starts, the goal is to keep safety violations at zero while maximizing return; in infeasible starts, the goal is to minimize harm. The results show several consistent patterns. On the double integrator and the Safety Gym task, FAITH achieves the highest return among methods that never violate safety from feasible initial states. That is important because it suggests the filter is not just being overly conservative; it preserves performance better than competing safe methods while still maintaining safety. From infeasible starts, FAITH matches the lowest harm achieved by the best methods, which means the same framework also behaves appropriately when safety is impossible. The humanoid results are especially interesting because they test the high-dimensional regime the paper is targeting. In Walking-Avoid, FAITH reaches a 99.95% safety rate while retaining 97% of the unfiltered return. That is a strong sign that the filter is not destroying the task behavior. In Push-Avoid, it obtains the highest measured safety rate by learning to sacrifice balancing and fall away from the protected region. That behavior is a good example of feasibility-aware control: when the robot can no longer stay fully upright and safe, it chooses a strategy that reduces harm rather than stubbornly trying to preserve nominal locomotion. The fact that the same policy is then demonstrated on a physical humanoid suggests the approach is not limited to simulation, although the paper presents these as qualitative demonstrations rather than a full hardware benchmark. What should we take away from all of this? The main contribution is not just a new safety filter, and not just a new safe RL objective. It is the combination of a learned safety critic, a feedforward approximation to minimal-intervention filtering, and training the task policy through that filter so that safety no longer competes directly with reward in the policy update. That makes the method especially appealing for high-dimensional systems where online optimization is expensive and where infeasible states are unavoidable in practice. There are still limitations to keep in mind. The safety critic is learned, so it is only as good as the data and function approximation allow; the paper explicitly treats it as a practical estimate rather than a formal certificate. The theoretical guarantees depend on idealized assumptions like a perfect filter and global optimization, which real neural networks will not satisfy exactly. And because the method relies on a learned acceptance condition, its behavior near the boundary of feasibility will depend on how accurately the critic captures the true safety landscape. So the next questions are about robustness: how stable is the method under larger distribution shifts, how sensitive is it to critic error, and how far can the feedforward approximation be pushed before it needs a stronger online correction mechanism? Even with those open questions, FAITH makes a compelling case that feasibility-aware safety filtering is a promising direction for safe control of complex robots.