User Model Extraction via Belief Self-Distillation
Ali Holmov, Yiran Huang, Kirill Bykov, Zeynep Akata
arXiv:2609.31603v1Large language models don’t just answer our questions; they also build hidden beliefs about who we are, and those beliefs can shape how they respond. This paper asks a simple but important question: can we inspect and even change a model’s internal user model, instead of only probing its surface behavior? The authors introduce Belief Self-Distillation, a method where the frozen model teaches itself a compact representation of user attributes from ordinary conversations, without labels or outside supervision. That representation is designed to be both readable and writable, so researchers can test whether changing the model’s belief actually changes its decisions. They show that it does: altering inferred user intent can shift refusal behavior while keeping the request itself fixed. The broader significance is big for AI safety and transparency. If models are quietly adapting to their beliefs about us, we need ways to audit those beliefs, understand them, and control them.
Also spotted that day
Previous daily papers