好文集ChatGPT Plus¥119

Finding the Reward Hacking Vector in Weight Space

文章目录

部分视频或图片暂未收录,已在正文保留提示;需要时可查看原文。查看原文 ↗

Reward hacking happens when a model learns to satisfy the reward, but not the behavior we actually want. This is a notorious challenge in RL for LLM.

One common remedy is to inspect bad trajectories with your favorite agent, identify unwanted behaviors, and make the reward stricter.

But what if I told you we could find a steering vector for this unwanted behavior in model's weight?

With this vector, we can turn the behavior up or down. We can even amplify or suppress it beyond what RL training achieves.

I will show you how.

文章配图

Do you want to bring chaos or bring order?

A controlled reward hacking

We study a controlled form of it in competitive programming RLVR: A brute-force solution can pass every small-input test and still time out on larger inputs.

We then gradually relax the reward, make it easier to hack, by dropping unit tests and these brute-force solutions receive positive reward.

The rewards are nested by design: if a solution passes strict reward, it passes the loose reward.

[未转换内容,参见原始 JSON]

[未转换内容,参见原始 JSON]

During the eval, we still use the full eval suite. We deliberately create this training-eval mismatch to mechanically catch reward hacking. Now we are studying reward hacking in the RLVR territory.

Finding the Reward Hacking Vector

We train independent RL models from the same SFT checkpoint, using the same data and hyperparameters.

The ONLY difference is how strict the reward is. And therefore, different trajectories are reinforced, and these differences are baked into the model weights.

Now the surprising part, we show that this difference lies in the linearly-connected basin.

Suppose your have 2 RL checkpoint, in which loose-reward checkpoint (checkpoint A) hacks more often and strict-reward checkpoint (checkpoint B) hacks less often, you can simply do 0.5 × A + 0.5 × B to land a model in the middle.

[未转换内容,参见原始 JSON]

BTW, this is called model merging and is an established field, such as "linear mode connectivity", "task vector", "reward soup" etc.

But here, the weight difference gives us something particularly interesting: a direction that controls the reward-hacking behavior. Or we call this its steering vector.

But why stop at interpolation?

We can positively extrapolate to amplify this behavior, or negatively extrapolate to suppress even this behavior.

文章配图

Yes, we're literally doing arithmetic on model weights. This sounds like crazy. You can construct a model, 1.2 * A - 0.2 * B which hacks more often than A, or -0.2 * A + 1.2 * B which hacks even less than B.

What’s the fun here?

Making a reward stricter can be difficult. Making it easier to hack is much simpler.

We can amplify the unwanted behavior through RL, then use the resulting weight-space direction to suppress it beyond what the strict-reward model achieves, without any training.

Extrapolate to suppress it

And it actually works.

The strict-reward model already reduces resource-limit failures from 22.8% to 14.5%. Extrapolation pushes this further down to 8.0%, without another RL run.

文章配图

And this is not just about fewer timeouts. Extrapolation also makes the generated programs faster. On harder-to-run problems, 71% of the commonly solved programs run faster.

文章配图

The Correctness Tax

Counter-intuitively, making the reward stricter doesn't necessarily make the model better.

Across independent RL runs, we observe that stricter rewards come with a “correctness tax”: it reduces resource-limit failures, but increases correctness failures on hard problems. The overall solve rate barely changes.

Of course, we can't extrapolate forever.

If we push extrapolation too far, the model eventually starts producing malformed outputs, like repetition and never stops the reasoning.

文章配图

We can do extrapolation up to a crazy range, but eventually the model output collapse.

Ending

These are the interesting bits from our paper. And luckily, it’s accepted at NeurIPS!

As I said in the abstract of the paper, this suggests a simple intervention for this form of reward hacking.

  1. Amplify: Use a loose reward to amplify the undesired behavior

  2. Identify: Use the weight-space difference to a strict-reward model as its steering vector

  3. Suppress: Extrapolate past the strict model to suppress it

文章配图

Looking back, I had the first interpolation/extrapolation result back in January this year. It’s probably my last slow-research paper that took > 6 months to turn an accidental discovery into the paper you see today.

If you're curious, check out the full paper!

📃 Correctness vs Efficiency: Behavior Control with Weight-Space Extrapolation in Code RL https://arxiv.org/abs/2605.28751

全文完
查看原文