Empirical Trust-Signal Reward Hacking in Multi-Agent RL

Investigating whether RL agents rewarded on imperfect trust signals become trustworthy or learn to exploit the metric.

Status: Active Research

Started: June 19, 2026

Last Updated: July 3, 2026

Overview

Reinforcement learning systems are often optimized against proxy objectives rather than the underlying goals designers actually care about. This creates opportunities for reward hacking, where agents maximize rewards without achieving the intended outcome.

This research investigates whether agents rewarded on imperfect trust signals learn genuinely trust-building behaviour or learn to exploit discrepancies between latent trust and observable trust metrics. Trust serves as a controlled experimental setting for studying broader alignment failures caused by proxy rewards.

Research Question

Can reinforcement learning agents learn to manipulate trust metrics instead of becoming trustworthy when rewarded on imperfect trust signals?

Key Questions

  • Under what conditions do agents exploit trust metrics rather than increase underlying trust?
  • How large must the gap between latent trust and reported trust be before exploitation emerges?
  • Does reward hacking emerge gradually or as an abrupt behavioural transition?
  • How does increasing agent capability affect the likelihood of exploitation?
  • Can reward design or regularization reduce proxy exploitation?
  • Are some trust-reporting mechanisms more robust to manipulation than others?

Current Hypotheses

  • H1: Proxy Exploitation — Agents rewarded on imperfect trust signals will learn policies that maximize reported trust without necessarily increasing underlying trust.
  • H2: Capability Thresholds — Reward hacking will become more pronounced as agent capability increases.
  • H3: Phase Transition Behaviour — Exploitation will emerge abruptly after crossing a capability threshold rather than appearing gradually.
  • H4: Robust Signals — Trust metrics incorporating longer historical context will be harder to exploit than heavily recency-weighted signals.

Approach

  • Construct a multi-agent environment containing a latent trust state, a reported trust signal, and controllable reporting bias.
  • Train agents using PPO.
  • Compare latent trust trajectories against reported trust trajectories.
  • Measure divergence between genuine trust-building and reward-maximizing behaviour.
  • Evaluate different levels of reporting bias and agent capability.

Experimental Variables

Independent Variables

  • Degree of reporting bias
  • Trust signal design
  • Agent capability
  • Training duration

Dependent Variables

  • Latent trust
  • Reported trust
  • Reward obtained
  • Trust-report divergence
  • Emergence point of exploitative behaviour

Notes & Ideas

  • Trust signals resemble many real-world feedback systems, including ratings, reputation scores, and human preference signals.
  • Reward hacking may initially appear beneficial because reported trust continues increasing while latent trust deteriorates.
  • The divergence between latent and reported trust may serve as an early warning signal for alignment failures.

Open Problems

  • How should latent trust be modeled realistically?
  • How can exploitation be distinguished from strategic cooperation?
  • What metrics best quantify trust-report divergence?
  • Are observed behaviours specific to trust environments or representative of proxy optimization more generally?